Graphing a system of inequalities in 3 dimensions

With luadraw, the 3d window is a parallelepiped (g:Box3d()) which can be cut with planes, so there are no calculations to be done:

\documentclass[border=5pt]{standalone}% compile with lualatex only
\usepackage[svgnames]{xcolor}
\usepackage[3d]{luadraw}%https://github.com/pfradin/luadraw
\usepackage{fourier-otf}
%https://tex.stackexchange.com/questions/761804/graphing-a-system-of-inequalities-in-3-dimensions
\begin{document}
\begin{luadraw}{name=q761804}
local g = graph3d:new{ adjust2d=true, viewdir=perspective("central",40,60,20), size={12,12}}
Hiddenlinestyle = "dashed"
local P1, P2, P3 = planeEq(-0.5,1,-1,0), planeEq(1,0,-1,0), planeEq(1,1,-3,0)
local A, cube = M(1,1,0), g:Box3d()
for _, P in ipairs({P1,P2,P3}) do
    if pt3d.dot(A-P[1], P[2]) < 0 then P[2] = -P[2] end
    cube = cutpoly(cube,P,true)
end
g:Dboxaxes3d({grid=true, gridcolor="gray", fillcolor="lightgray", drawbox=true})
g:Dscene3d(
    g:addPlane(P1, {color="ForestGreen", edge=true, opacity=0.8, scale=1, contrast=0}),
    g:addPlane(P2, {color="Gold", edge=true, opacity=0.8, scale=1, contrast=0}),
    g:addPlane(P3, {color="Navy", edge=true, opacity=0.8, scale=1, contrast=0}),
    g:addPoly(cube, {color="Pink", edge=true, hidden=true}) )
g:Dlabel("$P_1$",Z(4.5,5),{pos="NW"}, "$P_2$",Z(-6.5,4),{pos="N"},"$P_3$",Z(6.5,0.5),{pos="N"})
g:Dlabel3d( "Solutions", M(2,5,-1), {dir={-vecI,vecK},pos="N"})
g:Show()
\end{luadraw}
\end{document}

The original problem defines an unbounded region; for visualization, we add the cube Q=[0,4]^3. Next, we compute the vertices of the polyhedron bounded by the three planes and contained in the cube Q.

On the face z=0, the inequalities reduce to y\ge x/2, so we obtain the four vertices (0,0,0), (0,4,0), (4,4,0), and (4,2,0). On the face y=4, the intersections z=x=(x+y)/3 and z=y-x/2=(x+y)/3 give the vertices (2,4,2) and (16/5,4,12/5). Finally, on the edge x=y=4, we have z\le \min(4-4/2, 4, (4+4)/3)=2, which gives the vertex (4,4,2). Thus the vertices are (0,0,0), (0,4,0), (4,4,0), (4,2,0), (2,4,2), (16/5,4,12/5), and (4,4,2).

In this picture, the green and pink faces are not parts of the original three planes; they are faces of the auxiliary cube used to truncate the unbounded region. More precisely, the green face lies in the plane y=4, and the pink face lies in the plane x=4. The blue bottom face is also a face of the cube, namely z=0. By contrast, the three upper faces belong to the planes z=x, z=(x+y)/3, and z=y-x/2, which come from the original system of inequalities.

\documentclass[a4paper,12pt]{article}
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}

\begin{document}

\begin{tikzpicture}
\begin{axis}[
    width=12cm,
    view={125}{45},
    xmin=0, xmax=4,
    ymin=0, ymax=4,
    zmin=0, zmax=4,
    xlabel={$x$},
    ylabel={$y$},
    zlabel={$z$},
    axis lines=box,
    grid=major,
    ticks=both,
    xticklabels={,0,1,2,3,},
    zticklabels={,,2,4},
]

% Vertices of the polyhedron:
% A=(0,0,0)
% B=(0,4,0)
% C=(2,4,2)
% D=(3.2,4,2.4)
% E=(4,4,2)
% F=(4,4,0)
% G=(4,2,0)

% Base face z=0
\path[draw=blue!60!black, fill=blue!30, fill opacity=0.20]
(axis cs:0,0,0) --
(axis cs:0,4,0) --
(axis cs:4,4,0) --
(axis cs:4,2,0) -- cycle;

% Side face x=4
\path[draw=blue!60!black, fill=red!30, fill opacity=0.20]
(axis cs:4,2,0) --
(axis cs:4,4,0) --
(axis cs:4,4,2) -- cycle;

% Side face y=4
\path[draw=blue!60!black, fill=green!30, fill opacity=0.50]
(axis cs:0,4,0) --
(axis cs:2,4,2) --
(axis cs:3.2,4,2.4) --
(axis cs:4,4,2) --
(axis cs:4,4,0) -- cycle;

% Upper face in the plane z=x
\path[draw=blue!60!black, fill=blue!60, fill opacity=0.7]
(axis cs:0,0,0) --
(axis cs:0,4,0) --
(axis cs:2,4,2) -- cycle;

% Upper face in the plane z=(x+y)/3
\path[draw=blue!60!black, fill=yellow!50, fill opacity=0.35]
(axis cs:0,0,0) --
(axis cs:2,4,2) --
(axis cs:3.2,4,2.4) -- cycle;

% Upper face in the plane z=y-x/2
\path[draw=blue!60!black, fill=blue!40, fill opacity=0.35]
(axis cs:0,0,0) --
(axis cs:4,2,0) --
(axis cs:4,4,2) --
(axis cs:3.2,4,2.4) -- cycle;

% Highlight the upper edges
\addplot3[very thick, blue!70!black] coordinates {
(0,0,0)
(2,4,2)
(3.2,4,2.4)
(4,4,2)
};

\addplot3[very thick, blue!70!black] coordinates {
(0,0,0)
(4,2,0)
(4,4,2)
};

\end{axis}
\end{tikzpicture}

\end{document}

1. As the error says: You have to run pdflatex with the option --shell-escape, to allow pdflatex run shell commands itself.

2. With this command-line option, we run into the error undefined control sequence \pgfmath@. The reason seems to be that you use \frac in the expressions specifying the planes. \frac is for typesetting fractions. In the expressions, use / for division.

See below for the TeX source.

Compile the document with pdflatex --shell-escape.

\documentclass[12pt, a4paper]{article}
\usepackage{float, ulem, amsmath, amsthm, amssymb, pgfplots, tikz}
\pgfplotsset{width=10cm,compat=1.9}
\usepgfplotslibrary{external, fillbetween}
\tikzexternalize 

\begin{document}

\begin{tikzpicture} 
  \begin{axis}[domain=0:10,y domain=0:10]
    \addplot3[surf] {y-(0.5*x)};
    \addplot3[surf] {x};
    \addplot3[surf] {(x+y)/3};
  \end{axis}
\end{tikzpicture}

\end{document}