How do you visualize neural network architectures?

Question

When writing a paper / making a presentation about a topic which is about neural networks, one usually visualizes the networks architecture.

What are good / simple ways to visualize common architectures automatically?

score 98 · Answer 1 · edited Aug 29 '22 at 01:27

98

I recently created a tool for drawing NN architectures and exporting SVG, called NN-SVG

edited Aug 29 '22 at 01:27

Christoph Rackwitz

103
4

answered May 10 '18 at 14:41

Alex Lenail

1,081
7
2

Martin Thoma · Answer 2 · 2019-09-23T08:55:47.800

Tensorflow, Keras, MXNet, PyTorch

If the neural network is given as a Tensorflow graph, then you can visualize this graph with TensorBoard.

Here is how the MNIST CNN looks like:

You can add names / scopes (like "dropout", "softmax", "fc1", "conv1", "conv2") yourself.

Interpretation

The following is only about the left graph. I ignore the 4 small graphs on the right half.

Each box is a layer with parameters that can be learned. For inference, information flows from bottom to the top. Ellipses are layers which do not contain learned parameters.

The color of the boxes does not have a meaning.

I'm not sure of the value of the dashed small boxes ("gradients", "Adam", "save").

score 33 · Answer 3 · edited Jun 16 '20 at 11:08

There is an open source project called Netron

Netron is a viewer for neural network, deep learning and machine learning models.

Netron supports ONNX (.onnx, .pb), Keras (.h5, .keras), CoreML (.mlmodel) and TensorFlow Lite (.tflite). Netron has experimental support for Caffe (.caffemodel), Caffe2 (predict_net.pb), MXNet (-symbol.json), TensorFlow.js (model.json, .pb) and TensorFlow (.pb, .meta).

score 24 · Answer 4 · answered Mar 27 '18 at 16:04

I would add ASCII visualizations using keras-sequential-ascii (disclaimer: I am the author).

A small network for CIFAR-10 (from this tutorial) would be:

       OPERATION           DATA DIMENSIONS   WEIGHTS(N)   WEIGHTS(%)

           Input   #####     32   32    3
          Conv2D    \|/  -------------------       896     2.1%
            relu   #####     30   30   32
    MaxPooling2D   Y max -------------------         0     0.0%
                   #####     15   15   32
          Conv2D    \|/  -------------------     18496    43.6%
            relu   #####     13   13   64
    MaxPooling2D   Y max -------------------         0     0.0%
                   #####      6    6   64
         Flatten   ||||| -------------------         0     0.0%
                   #####        2304
           Dense   XXXXX -------------------     23050    54.3%
         softmax   #####          10

For VGG16 it would be:

       OPERATION           DATA DIMENSIONS   WEIGHTS(N)   WEIGHTS(%)

          Input   #####      3  224  224
     InputLayer     |   -------------------         0     0.0%
                  #####      3  224  224
  Convolution2D    \|/  -------------------      1792     0.0%
           relu   #####     64  224  224
  Convolution2D    \|/  -------------------     36928     0.0%
           relu   #####     64  224  224
   MaxPooling2D   Y max -------------------         0     0.0%
                  #####     64  112  112
  Convolution2D    \|/  -------------------     73856     0.1%
           relu   #####    128  112  112
  Convolution2D    \|/  -------------------    147584     0.1%
           relu   #####    128  112  112
   MaxPooling2D   Y max -------------------         0     0.0%
                  #####    128   56   56
  Convolution2D    \|/  -------------------    295168     0.2%
           relu   #####    256   56   56
  Convolution2D    \|/  -------------------    590080     0.4%
           relu   #####    256   56   56
  Convolution2D    \|/  -------------------    590080     0.4%
           relu   #####    256   56   56
   MaxPooling2D   Y max -------------------         0     0.0%
                  #####    256   28   28
  Convolution2D    \|/  -------------------   1180160     0.9%
           relu   #####    512   28   28
  Convolution2D    \|/  -------------------   2359808     1.7%
           relu   #####    512   28   28
  Convolution2D    \|/  -------------------   2359808     1.7%
           relu   #####    512   28   28
   MaxPooling2D   Y max -------------------         0     0.0%
                  #####    512   14   14
  Convolution2D    \|/  -------------------   2359808     1.7%
           relu   #####    512   14   14
  Convolution2D    \|/  -------------------   2359808     1.7%
           relu   #####    512   14   14
  Convolution2D    \|/  -------------------   2359808     1.7%
           relu   #####    512   14   14
   MaxPooling2D   Y max -------------------         0     0.0%
                  #####    512    7    7
        Flatten   ||||| -------------------         0     0.0%
                  #####       25088
          Dense   XXXXX ------------------- 102764544    74.3%
           relu   #####        4096
          Dense   XXXXX -------------------  16781312    12.1%
           relu   #####        4096
          Dense   XXXXX -------------------   4097000     3.0%
        softmax   #####        1000

Franck Dernoncourt · Answer 5 · 2020-12-29T23:41:57.220

22

In Caffe you can use caffe/draw.py to draw the NetParameter protobuffer:

In Matlab, you can use view(net)

Keras.js:

Also, see Can anyone recommend a Network Architecture visualization tool? (Reddit/self.MachineLearning).

edited Dec 29 '20 at 23:41

answered Jul 19 '16 at 00:43

Franck Dernoncourt

5,862
12
44
80

score 19 · Answer 6 · answered Jan 22 '18 at 10:48

Keras

The keras.utils.vis_utils module provides utility functions to plot a Keras model (using graphviz)

The following shows a network model that the first hidden layer has 50 neurons and expects 104 input variables.

plot_model(model, to_file='model.png', show_shapes=True, show_layer_names=True)

score 14 · Answer 7 · answered Oct 12 '20 at 09:27

PlotNeuralNet LaTex tool

This solution is not automatically generated (you need to construct the graph by yourself) but the PlotNeuralNet github repo allows you to build images directly from LaTex, and the result is great ! See for example the image below from the README :

or my example :

score 13 · Answer 8 · answered Dec 11 '17 at 12:33

13

Here is yet another way - dotnets, using Graphviz, heavily inspired by this post by Thiago G. Martins.

dotnets example

answered Dec 11 '17 at 12:33

bytesinflight

231
2
4

Jesse · Answer 9 · 2020-01-29T15:18:45.327

13

I've been working on a drag-and-drop neural network visualizer (and more). Here's an example of a visualization for a LeNet-like architecture. Models with fan-out and fan-in are also quite easily modeled. You can visit the website at https://math.mit.edu/ennui/

The open-source implementation is available at https://github.com/martinjm97/ENNUI.

edited Jan 29 '20 at 15:18

answered Apr 09 '19 at 19:52

Jesse

231
2
4

score 12 · Answer 10 · answered Mar 05 '18 at 12:34

The Python package conx can visualize networks with activations with the function net.picture() to produce SVG, PNG, or PIL Images like this:

Conx is built on Keras, and can read in Keras' models. The colormap at each bank can be changed, and it can show all bank types.

More information can be found at: http://conx.readthedocs.io/en/latest/

score 10 · Answer 11 · answered Jul 21 '16 at 17:32

In R, nnet does not come with a plot function, but code for that is provided here.

Alternatively, you can use the more recent and IMHO better package called neuralnet which features a plot.neuralnet function, so you can just do:

data(infert, package="datasets")
plot(neuralnet(case~parity+induced+spontaneous, infert))

neuralnet is not used as much as nnet because nnet is much older and is shipped with r-cran. But neuralnet has more training algorithms, including resilient backpropagation which is lacking even in packages like Tensorflow, and is much more robust to hyperparameter choices, and has more features overall.

score 7 · Answer 12 · edited Jul 03 '18 at 15:10

There are some novel alternative efforts on neural network visualization.

Please see these articles:

Stunning 'AI brain scans' reveal what machines see as they learn new skills

Inside an AI 'brain' - What does machine learning look like?

These approaches are more oriented towards visualizing neural network operation, however, NN architecture is also somewhat visible on the resulting diagrams.

Examples:

score 6 · Answer 13 · answered Oct 05 '20 at 09:56

Tensorflow / Keras / Python

I wrote a small python package called visualkeras that allows you to directly generate the architecture from your keras model.

Install via pip install visualkeras

And then it's as simple as:

import visualkeras
visualkeras.layered_view(<model>)

There are lots of options to tweak it and I am working on more visualizations. Also, always open for PRs or feature requests.

Here's what VGG16 looks like:

score 5 · Answer 14 · answered Jan 22 '18 at 12:00

You can read the popular paper Understanding Neural Networks Through Deep Visualization which discusses visualization of convolutional nets. Its implementation not only displays each layer but also depicts the activations, weights, deconvolutions and many other things that are deeply discussed in the paper. It's code is in caffe'. The interesting part is that you can replace the pre-trained model with your own.

Ali Mirzaei · Answer 15 · 2019-01-26T14:43:12.020

5

Tensorspace-JS is a fantastic tool for 3d visualization of network architecture:

https://tensorspace.org/

and here is a nice post about how to write a program:

https://medium.freecodecamp.org/tensorspace-js-a-way-to-3d-visualize-neural-networks-in-browsers-2c0afd7648a8

edited Jan 26 '19 at 14:43

answered Jan 25 '19 at 07:56

Ali Mirzaei

151
1
4

Thomas Wagenaar · Answer 16 · 2017-05-15T18:41:41.730

4

Not per se nifty for papers, but very useful for showing people who don't know a lot of about neural networks what their topology may look like. This Javascript library (Neataptic) lets you visualise your network:

edited May 15 '17 at 18:41

answered Apr 10 '17 at 08:22

Thomas Wagenaar

1,158
8
7

score 4 · Answer 17 · answered Jan 25 '19 at 13:31

4

Netscope is my everyday tool for Caffe models.

answered Jan 25 '19 at 13:31

Dmytro Prylipko

856
5
10

score 2 · Answer 18 · answered May 29 '22 at 07:15

2

I have found one amazing website. You just need to upload your h5 model, Then you will get a beautiful visualization within a few seconds. Check it out!

answered May 29 '22 at 07:15

Aravind R

136
3

score 1 · Answer 19 · answered Mar 19 '23 at 10:30

For a solution for PyTorch I'd add TorchView.

It is as easy as:

from torchview import draw_graph
model = MLP()
batch_size = 2
device='meta' -> no memory is consumed for visualization
model_graph = draw_graph(model, input_size=(batch_size, 128), device='meta')
model_graph.visual_graph

Which yields:

It has many customization options as well.

psychicmachinist · Answer 20 · 2023-12-15T08:52:26.853

I'll add a plug for my recent project, TorchExplorer (live demo here). It's sort of a combination of Netron and wandb.watch. It can:

Interactively traverse model architectures, showing input/output tensor sizes and module parameters
Visualize module input/output tensors, parameters, and associated gradients as histograms over the course of training (modeled off of wandb.watch)
Directly integrate with weights and biases or serve standalone with a simple torchexplorer.watch call

Eduardo Lozano · Answer 21 · 2020-06-12T00:05:35.960

You can use eiffel2, which you can install using pip:

python -m pip install eiffel2

Just import builder from eiffel and provide a list of neurons per layer in your network as an input.

Example:

from eiffel2 import builder

builder([1, 10, 10, 5, 5, 2, 1])
# or the following if you want to have a dark theme
builder([1, 10, 10, 5, 5, 2, 1], bmode="night")

Output:

To see more about eiffel2 visit the Github repository:

https://github.com/Ale9806/Eiffel2/blob/master/README.md

How do you visualize neural network architectures?

21 Answers21

Tensorflow, Keras, MXNet, PyTorch

Interpretation

PlotNeuralNet LaTex tool

Tensorflow / Keras / Python

device='meta' -> no memory is consumed for visualization

Linked

Related