Here are the connecting arcs between your 4 (red) line segments :
Fig. 1 : "Your" line segments (in red) are connected by arcs of hyperbolas. They have been extended in order to understand them in a better way (otherwise, one could think that for example arc $MN$ is a linear segment). Point $M$ has coordinates $(85/3 ; 25)$.
Edit : in fact, after a first answer, I have realized that there is a much better way to understand and solve this issue. This is what is presented as the first solution. The initial answer has been "downgraded" as a second solution.
First solution :
The goal of the study can be changed into the more general question : "Obtain an expression of the area of the rectangle $R$, intersection of the two squares $B$ and $S$" (Big and Small)", then restrain this area to be at least $800$ (exercise : why this number ?).
Let us consider the big square $B$ as fixed in a coordinate system where its vertices are $(\pm20,\pm20)$ and the second square $S$ with variable center in $(x,y)$, and vertices in $(x\pm15,y\pm15)$.
Instead of considering at once "squares' overlap", let us consider the simpler case of "strips' overlap" : being given two strips of paper with the same width, say $L$ (the value of $L$ is unimportant) ; in the middle of the first, resp. second, strip, a $L' \times 40$, resp. $L'\times 30$ rectangular window is cutted ($L'<L$). If we slide one of the strips along the other, consider the variable amount of light passing through the intersection. Let us call $f$ this function ; its graphical representation is given on fig. 2.*
Fig. 2 : Function $f$ as described above. Its different parts have equations $f(x)=0, \ \ f(x)=35 \pm x, \ \ f(x)=30$.
Remark : (can be skipped in a first reading) $f$ is in fact the convolution of 2 constant functions with resp. support length $40$ and $30$.
As a consequence (see legend of Fig. 2), the area of the intersection rectangle $R$ is $z=f(x)f(y)$, represented on Fig. 3 as a surface looking like a Maya pyramid... with :
a summital square at "altitude" $900 = 30 \times 30$ ; indeed, the different points of this square correspond to the positions of the center $(x,y)$ of the small square $S$ when it is fully included into the big square $B$,
$4$ planar "ramps" (planes with equations $z=35 \pm x$ or $z=35 \pm y$, and
$4$ skew surfaces. One of them (deep blue color) is a quadric surface with equation $z=(35-x)(35-y)$, due to the fact that $f(x)=35-x$ for $5 \le x \le 35$.
As a consequence, the upper-left level curve will be obtained by writing :
$$(35-x)(35-y)=800 \ \ \iff \ \ y=35-\frac{800}{35-x}$$
which is the equation of the top right hyperbola in Fig. 1.
Fig. 3 : Graphical representation of $z=f(x)f(y)$ where $z$ is the value of the surface of the common rectangle of the two squares when the center of the small square is positionned in $(x,y)$.
Fig. 4 : "Level lines" of the surface represented in Fig. 3. Level line $z=800$ (the smallest one, just surrounding the "summital square" at $z=900$) is the one which is represented on Fig. 1.
See "Sage" program below for the generation of figures $2,3,4$.
Second solution (initial answer) :
Let $D$ be the "difference set" of points belonging to the big square but not to the small square.
$D$ has three possible shapes :
You have studied the two last ones, resulting in the red frontier lines you have given.
The "L-case", up to a rotation, is represented in the following figure :
Fig. 5 : The two squares, with the non-covered "L-shape" area in red.
Let us show why, in this case, the boundary is a hyperbolic arc.
Adding the 3 red rectangular areas of Fig. 2, one gets :
$$A_{x,y}:=x(40-y)+y(40-x)+xy\tag{1}$$
Saying that the orange common area is at least $800 = 40 \times 40/2$ is equivalent to say that
$$A_{x,y} \le 800 \ \iff \ 40(x+y)-xy \le 800$$
The boundary curve has therefore the following equation:
$$40(x+y)-xy = 800 \ \iff \ y=40 \frac{20-x}{40-x}\tag{2}$$
which is the equation of a hyperbola (to be considered in a certain range of values of $x$, therefore giving only an arc of hyperbola).
"Sage" program for Fig. $2,3$ and $4$ (please note the need to use the suffix "symbolic" for max and min operators) :
var('a')
a=50;f(x)=max_symbolic(0,min_symbolic(35+x,35-x,30))
G=plot(f(x),(x,-a,a),thickness=2,aspect_ratio=1)
show(G)
g(x,y)=f(x)*f(y)
G=contour_plot(g(x,y), (-a,a), (-a,a), contours=(0,100,200,300,400,500,600,700,800,899.9), fill=False)
show(G)
G=plot3d(g(x,y),(x,-a,a),(y,-a,a), color='cyan',plot_points=250,aspect_ratio=[1,1,0.05])
G+=plot3d((-x+35)*(-y+35),(x,5,35),(y,5,35), color='blue',plot_points=250,aspect_ratio=[1,1,0.05])
show(G)
It is noteworthy that a different way to express $f$ function is to define first the ramp function (primitive function of heaviside function), then define $f(x)$ as a linear combination of translates of $r$ in the following way (please note that the sum of coefficients of the $r(x-a)$ is zero):
r(x)=max_symbolic(0,x)
f(x)=r(x+35)-r(x+5)-r(x-5)+r(x-35)
