It is said that triangle inequality for the space $L^p(\mathbb{R})$ space doesn't hold if $0<p<1$.
Does anyone know an example for this?
Also, what we can say, for example, about the quantity like $\| f \| \colon =\int_{\mathbb{R}} \sqrt{|f|} dx$?
I think in the space $\{ f ; \|f\| < \infty \}$, triangle inequality $\|f+g\| \le \|f\| + \|g\|$ is valid.
The reason that the Minkowski inequality (triangle inequality) fails for $\|f\|_p=\Big(|f|^p\Big)^{1/p}$ when $0<p<1$, is because $t\mapsto t^p$ on $[0,\infty)$ is not convex for $0<p<1$.
In fact, in this case the triangle inequality changes direction for all $f,g\geq0$. Indeed, assume without loss of generality that $0<\|f\|_p,\|g\|_p<\infty$. Let $t=\frac{\|f\|_p}{\|f\|_p+\|g\|_p}$. As the map $x\mapsto x^p$ is concave on $[0,\infty)$, $$ \Big(t\frac{f}{\|f\|_p}+(1-t)\frac{g}{\|g\|_p}\Big)^p\geq t\frac{f^p}{\|f\|^p_p}+(1-t)\frac{g^p}{\|g\|^p_p}$$ by Jensen's inequality. Integrating both sides of the inequality above yields $$ \frac{\int(f+g)^p\,d\mu}{\big(\|f\|_p+\|g\|_p\big)^p}\geq 1 $$ whence $$\|f+g\|_p=\Big(\int(f+g)^p\,d\mu\Big)^{1/p} \geq \|f\|_p+\|g\|_p$$
On the other hand, for $0<p<1$ the function $d:L_p\times L_p\rightarrow[0,\infty)$ given by $$d(f, g):=\|f-g\|^p_p=\int|f-g|^p\,d\mu$$ does define a metric on $L_p$. This follows from the inequality $(a+b)^p\leq a^p+b^p$ which holds for all $a,b\geq0$. Thus $$|f-g|^p\leq(|f-h|+|h-g|)^p\leq|f-h|^p+|h-g|^p$$ for all $f,g,h$. Notice, however, that $\|f\|^p_p=d(0,f)$ is not a norm.
It can be proven that $(L_p,d)$ is in fact a complete metric space and that $d$ is translation invariant. This make $L_p$ an $F$-space.
For $0<\alpha<1$ and a measure space $(\Omega,\mathcal E,\mu)$, the function $$f\mapsto\int_\Omega \lvert f\rvert^\alpha\,d\mu$$ is indeed subadditive, because the modulus is and the map $x\mapsto x^\alpha$ is subadditive on $[0,\infty)$. This is a consequence of this lemma:
Let $g:[0,\infty)\to [0,\infty)$ be concave function such that $g(0)=0$. Then, $g$ is subadditive.
Proof: Suppose $g(x)+g(y)< g(x+y)$ for some positive $x,y$. Then, $$\frac{g(x+y)-g(y)}{x}>\frac{g(x)}x=\frac{g(x)-g(0)}{x},$$against the hypothesis that the incremental ratio is decreasing in both variables.
Thus $\int_\Omega \lvert f+g\rvert^\alpha\,dx\stackrel{x^\alpha\text{ increasing}}\le\int_\Omega (\lvert f\rvert+\lvert g\rvert)^\alpha\,dx\stackrel{\text{lemma}}\le\int_\Omega\lvert f\rvert^\alpha+\lvert g\rvert^\alpha\,d\mu=\int_\Omega\lvert f\rvert^\alpha\,d\mu+\int_\Omega\lvert g\rvert^\alpha\,d\mu$
As for an instance where $\lVert f+g\rVert_\alpha>\lVert f\rVert_\alpha+\lVert g\rVert_\alpha$ for $\alpha\in(0,1)$, you can shape it from the counterexample to $v_\alpha(x,y):=(x^\alpha+y^\alpha)^{1/\alpha}$ being a norm on $\Bbb R^2$.
In that case, one notices that, since $\frac1\alpha>1$, $$v_\alpha(1,1)=2^{1/\alpha}>v_\alpha(1,0)+v_\alpha(0,1)=2$$
In the case of functions $f=1_{[0,1]}$ and $g=1_{[1,2]}$ will do just fine.
For any $0\leq p<\infty$ (thus in particular for $0<p<1$), if $f,g$ have disjoint supports, then one has $$ \|f+g\|_p=(\|f\|_p^p+\|g\|_p^p)^{1/p}. $$ (This can be proved easily by definition.)
On the other hand, suppose $a,b>0$ and $0<p<1$, one has $(a+b)^p>a^p+b^p$ (exercise!).
Therefore, if $E$ and $F$ are disjoint sets of positive finite measure and we set $a=\mu(E)^{1/p}$ and $b=\mu(F)^{1/p}$ (where $\mu$ denotes the underlying measure), then $$ \|1_E+1_F\|_p=(a^p+b^p)^{1/p}>a+b=\|1_E\|_p+\|1_F\|_p $$ which gives a desired counterexample.
