wrapping up

I have had a luxurious summer break and haven’t touched the FPGA video synth for almost two months.

I want to wrap up the project. I’ve just looked through the blog to see what I can take from it :

• I want to expose the FPGA synth in its current state (with minimal or no modifications) at TEI 25′ (didn’t get accepted! Applying to DIS 2025 though) and in Xiao’s space (she doesn’t want electronics there).
• I want to make a kind of research paper / presentation with the HDMI imbalance hack. This is the coolest, most unexpected thing I found and it absolutely requires the FPGA and knowledge of verilog. I don’t want to go further towards the engineering tasks I was engaged in, nor the superficial design activities.

Specificity, and its details, was one of the keys about writing good scripts for the Making of Sopranos documentary Wise Guy. For this reason I don’t think making a framebuffer and then seeing all the different ways I can modify a single input image really makes any sense, there are just too many possibilities and it is too general / too much of an engineering project. Remember this broken screen project with videos playing : https://haoni.art/spin-III.

I could see my series of “design research chapters” as being composed of a series of mini strategies :

• freezing a single computer in time by stretching it up (prepared plotter, prepared 3D prints)
• one input, many different interpretations because machines are idiosyncratic (prepared CRT controller)
• the world view of the computer program (Revit internal IDs)

The natural things to test would be :

• How can I mess with the HDMI color balance, what is the range of possibilities trying different imbalances ?
• How do different screens react ?
• Could this be a kind of hacking of various video protocols like VGA, DVI and HDMI ?

More thoughts on the state of the project :

There is something cheap about putting images on screens, it’s so easy and it’s so ubiquitous that it doesn’t feel special or difficult to do or in any way sacred. At the same time, following the project where it wants to go, I want to work with video because I can with my modest tech skill level.

My video experiments have taught me a few things :

• For abstract animations, bit patterns are cool for digital and simple filters and oscillators are cool for analog. In the spirit of the latter experiments, I made a series of hand-made, wooden-based, simple filters that seemed like a conclusion in themselves. Beyond these it risks getting into very common computer science art territory around math (fractals, etc.) that you can see on shadertoy and processing. From doing p5js workshops (and NO SCHOOL telling me that hardware workshops are always the most enjoyed) I can say that playing with hardware is just more fun than making animations with code.
• Taking iconic cinema as input immediately seems richer, because there is culture and history involved, though I can’t say exactly why I should be distorting a film from the 1960s. Using memory to store video clips in hardware was a breakthrough for me, the severe limitations of SRAM were a useful creative constraint. I haven’t yet been able to take in a video stream and modify it live with the FPGA filters though the VGA camera could be a good solution for that and could lead to new tests.
• On applications : I don’t appear to want to make a cute product : too much polishing, purely technical work, admin and marketing, not enough fun and “design research”. On the other hand, my situation at work allows me to spend money on boards and components so I feel I should take advantage of the opportunity to test ideas in physical form. Hardware video workshops are cool and can work; I’ll test my first technical FPGA workshop in January. I am not sure what to say about this project if I were trying to write a paper. I don’t know if I want to fully engage with the VJ community as I don’t feel part of the music scene. Therefore the project remains hovering between a DIY / hobbyist open-source hardware art project, which doubles as a project to learn technical things, as well as a curated set of images and animations of my “design research” posted on Instagram. In other words, it has to be visually and technically interesting to me.

***************

Reviewing 20th C artists, some notes :

• Some of Duchamp’s readymades were actually called “assisted readymades” if they were altered.
• Décollage like the artwork 122 rue du Temple is an example of ripped collage.
• I feel more aligned with the Bauhaus (Anni Albers, László Moholy-Nagy) than NY based conceptual or minimalist artists.
• Some quotes : Albers “Technique was acquired as it was needed and as a fondation for future attempts. Unburdened by any practical considerations, this play with materials produced amazing results, textiles striking in their novelty, their fullness of color and texture, and possessing often barbabic beauty”. She produced “prototypes for industrial production”.
• MOMA Highlights on Moholy-Nagy : Abstraction is rare and often boring in photography, more interesting is an unfamiliar configuration of form competes for our attention with the subject (between figurative and abstract). Photography is objective, and its unpredictability unveils fresh experiences. Photography challenges old habits of seeing  by showing very distant or very small things, or for looking up or down. Photography has revolutionized modern vision.

• Man Ray Rayographs : “It is impossible to say which plane of the picture are to be interpreted as existing closer or deeper in space. The picture is a visual invention, an image without a real-life model to which we can compare it.” His Rayographs were unpredictable pictorial adventure.
• MOMA Highlights on Stan Brakhage : “In Brakhage’s hands [directly manipulating film stock] became a way to maintain a direct experiential relationship with his chosen medium while refining his art to the essential elements.”
• Add to the video reel : Eisenstein, Man with a movie camera, voyage à la lune, raging bull.

**************

There is another path forward, making an FPGA LVDS LCD controller board. I like that this would be following the logic of the project, going deeper and not running away from the challenge. On the other hand, I’m not convinced it gets much deeper than VGA or HDMI as I’m still dealing with rows and columns, H and V syncs, and colors. It is also potentially a technical hassle like the SD card headache.

You can already see some new colour and texture effects in this DIY project (https://community.element14.com/products/devtools/avnetboardscommunity/b/blog/posts/driving-a-laptop-lcd-using-an-fpga) :

Here are some resources for controlling a Flat panel display (AKA making a FPD-Link converter) with FPGA:

http://web.archive.org/web/20160109020747/http://g3nius.org/lcd-controller/

It looks like these are some examples of the typical signal setups (JEIDAVESA?) :

*****************

Yet another path forward is to complete the tests with the VGA camera input. Check out the pocket glitchomatic which has a camera (solves all the issues with taking video in!!!!)

and this camera called pixless on kickstarter :

This could work nicely with the code I was preparing where each combination of 8 bits selects different logic operations !

Circuit bent digital cameras by glitchedbyproxy :

Then again, couldn’t I just have a webcam on the rpi and output VGA ? No annoying configuration necessary for the camera ! Isn’t the real thing to learn here how to do convolution with the FPGA?

And some cool automata work here by Richard Palethorpe https://richiejp.com/1d-reversible-automata

********

OK, I am trying now to pick back up where I left off pre-summer, notably trying to save rpi video into SRAM through the FPGA in a stable way.

Trying to get the CC version of the board (with SRAM, buttons and RPI dock) back up to speed. I soldered VGA connector and added VGA pins to various .pcf had lying around. None appear to produce anything on the screen. EDIT : This was a pin constraints error. Looked on the scope and the pins weren’t doing anything. It looked at only one of the three input verilog files and ignored the pins from the others. ALWAYS CHECK THE PIN CONSTRAINTS, EVEN IF YOU MAKE A PCF !! This is the issue where IceCube2 can’t seem to deduce the hierarchy of the files and determine which one is TOP. Look at the file name of the bitmap, it should be the top file. Also, be suspicious if it takes too little time to synthesize ! The second issue I had was the vga_sync module not using the incoming clock (instead taking an output from a pll that had since been removed), so nothing was happening.

The rpi is now outputting 640×480 in DPI mode @ 32MHz, I checked on the scope. I think the easiest would be to get the rpi at the same timing. I would like to try again to run custom timings as this appears easier than modifying my fragile FPGA verilog code. (I have forgotten how to get a PLL running). EDIT : I am looking at how the Alhambra sets up a PLL and I’m just editing his pll file with different values : https://github.com/imuguruza/alhambra_II_test/blob/master/vga/vga_test/pll.v. This appears to work stably for turning 12MHz to 25MHz but when I try 100MHz to 32 MHz I run into issues (though perhaps I should just divide the clock in verilog?). I would love to get this rpi DPI calculator working https://nerdhut.de/software/raspberry-pi-dpi-calculator/ or learn to use the new way rpi encodes the screen timings. EDIT : got the rpi DPI calculator installed :

Unfortunately the first try (above) hasn’t yet worked on the rpi (it doesn’t show any pin activity on ‘scope). The best thing working is still the 640×480 at 32MHz with this end of the config :

gpio=0-9=a2
gpio=12-17=a2
gpio=20-25=a2
gpio=26-27=a2

dtoverlay=dpi24
enable_dpi_lcd=1
display_default_lcd=1
dpi_group=2
dpi_mode=87
dpi_output_format=516118
dpi_timings=640 1 44 2 42 480 1 16 2 14 0 0 0 60 0 32000000 1

So how can I divide my FPGA clock from 100MHz to 32MHz ? Tried creating a counter that counted up to 64 MHz and toggled another counter, but I didn’t get any video output (too lazy to debug).

I have a new timing which works (at least on the ‘scope) on the rpi to output 50MHz 800 x 600 :

So here are the two important lines :

dpi_output_format=0x17

dpi_timings=800 0 56 120 64 600 0 37 6 23 0 0 0 72 0 50000000 1

Now I will try to get the FPGA to output these timings…And it works at 800×600 @ 50MHz !

So here are the modifications I made to the vga_sync_test file :

localparam h_pixel_max = 800;

localparam v_pixel_max = 600;

localparam h_pixel_half = 400;

localparam v_pixel_half = 300;

...

// for a 50MHz clock we divide 100MHz by 2

reg [1:0] clk_div = 0;

always @(posedge clk_in)

begin

clk_div <= clk_div + 1;

end
...


vga_sync vga_s(

.clk_in(clk_div[0]), //Dividing by two gives 50MHz

.h_sync(h_sync),

.v_sync(v_sync),

.h_count(h_count),

.v_count(v_count),

.display_en(display_en) // '1' => pixel region

);

endmodule

And the modifications to the vga_sync file :

***********

There are some reasons to remake the CC SRAM board. I guess this shows I am doing incremental design, probably with the hope that this could be a product still one day. Perhaps I should tell myself that I am making a single version change per year ?  :

• The spot where the rpi plugs in blocks the programming pins and the SD card hits the VGA connector.
• A reset button !!!
• 2 pin DPI switch to load different images into FLASH
• Sending rpi_pixel clock to a BUFFERED input on the FPGA (like I am currently doing with the 100MHz clock to pin 49)
• A bigger (and FASTER, like 10ns) SRAM
• Two boards, one with keys, another with circuit. This gives me more space to add things, removes the constraints of the holes passing through the board, gives more buttons, tighter spacing (much more satisfying than the spaced out keys currently), gives it more heft, feels more like a real tool and less like a toy.
• The rpi needs its own power and the power plug can’t currently fit on when it is plugged in. Either I supply the power for the rpi or it is externally powered and I accomodate the plug.
• Currently I am only taking 4 pins from the rpi but ideally I would take 12 color input pins. I also should have h sync, v sync, and DEN wired directly from rpi to FPGA.
• Possibly add a track ball mouse ?
• It would be smart to look into how the rpi camera could be integrated into the design.
• Presumably I will want to move beyond 1280 logic gates to the 4K once I’ve done more tests ?
• Make it so that the Lattice programmer plugs in easily and can’t be plugged in wrong

Another insight : so much time is wasted setting up and resetting up the same setup. Having a permanent version of the screen + cable + rpi + FGPA combination with the wires not moving is going to be essential if I am to actually develop this further. I must switch to seeing this project as a long term one, and I must take my time and do things properly. I should program in reusable modules too !

How cool would it be to have an interface like this on the FPGA :

********

Check out these retro Tectronics screen images :

******

Just trying to make my life easier :

The rpi takes about 2 minutes to enter into video mode in DPI.

I am also being really careful each time I make something worth saving, because IceCube2 edits the files you load, I am making sure to copy and save the files (all verilog files + pcf) I want to keep and to name them so that it is clear what they do. With this structured approach I can build atop my previous work. EDIT : Tommy suggested the obvious : use a git ! Transitioning to this work flow now…Everything is going here :

https://github.com/preparedinstruments/bechamel/tree/main/verilog

I can record from the VGA into SRAM and keep the image mostly frozen. Here is the key line (that basically says only increment the SRAM address if we are in the active pixel area, and also divide it by 2 because the SRAM is only 256K and the total pixels are 480K, and restart the SRAM address counter at the end of the screen) :

always @(posedge clk_div[1]) begin

if (display_en) begin
if( addr < 17'b11111111111111111 && h_count > h_pixel_front_porch_amount && v_count > v_pixel_front_porch_amount && h_count < h_pixel_display && v_count < v_pixel_display && h_count[0]==1 && v_count[0]==1) begin // basically divide the counters because there are 480K pixels but I have 256K SRAM, and only count the active pixel area
addr <= addr+1;
end
else begin
addr <= 0; // I want to restart the counter at the end of each screen draw ?
end
end
end

It produces stable recordings, but they are imperfect :

input

output

So far it’s only working well with vertical lines though…EDIT fixed that !

Instead of incrementing the address, I am now taking the pixel position as the address.

addr <= h_count + (v_count*h_count);

Very noisy but the image is frozen solid with no movement ! But wait, this is wrong because it should be

addr <= h_count + (v_count*800);

EDIT : What seemed like success was in fact failure, I only had stable lines because I was recording only the first line and then resetting the SRAM addr to zero. I fixed that by only making the addr zero once I had reached the v_lines_max. I am getting sketchy recordings but I think it’s because the video is at 50MHz and the memory can’t go that fast?

********

Rereading these Project F FPGA graphics tutorials three things :

• You need to account for memory latency with an offset which you determine through testing when using a framebuffer
• A linebuffer saves you from doing too much memory access
• Shouldn’t calculate the pixel position by multiplying the horizontal_line_count + (vertical_line_count * horizontal_line_width) but should increment the addr value.

*****

I think to get recording w SRAM working I need to first record (and display, and clock the SRAM) with the input pixel clock + h&v syncs, and then switch to playback (and display, and SRAM clocking) with instead the FPGA board clock.

Another option might be to use PLL to synchronize external clocks that are out of phase somehow ? Also, if the clock and the incoming data are not synchronized, it is apparently not optional to double flop the input signal.

Some memory tests I’m preparing :

1. recover the simple design that captured rpi in and played back static images (even if they were distorted)
2. Do this but with BRAM.
3. Use rpi clock to record and display on screen, then switch to fpga clock to pb and display on screen.
4. Try #3 but with a linebuffer not a screen buffer.

Also check out this coding style guideline from nandland which suggests using these prefixes for verilog :

    i_   Input signal 
    o_   Output signal 
    r_   Register signal (has registered logic) 
    w_   Wire signal (has no registered logic) 
    c_   Constant 

Just visited the beautiful greenhouses in Glasgow and saw so many funky patterns and forms in the plants. It made me think about how form is nature, and exploring forms is natural too. I was struck by gradients, fractal patterns, by weirdness, scales of patterns, and the variety of different patterns.

****

Looking through some old video handbooks, here is an image showing the different video protocols on a map :

This shows the behind the screen blanking zones but repositioned to be in the middle of the screen :

**** 

At Christmas Owen gave me some super references and reminded me about the modular synth electronic music world and how they are into analog hardware that produces cool visuals.

The Yoto Player super low res screen and cartridges :

From https://www.peterzimon.com/hog/ :

Here’s my attempt at an interface for a 4×4 version :

Check out this great video about image processing in P5.js :

****

I’ve been trying to develop a simple, tangible AI workshop for designers / artists but so far it looks impossible to do in the way that I want.

The huggingface series is nice, not too technical. The machine vision series has some good background.

********

A great video from Coding Train on Convolutional Neural Nets https://www.youtube.com/watch?v=pRWq_mtuppU

The code is here : https://editor.p5js.org/codingtrain/sketches/GMRfsK7Wn

and also here for Perceptrons : https://www.youtube.com/watch?v=ntKn5TPHHAk

And this is a great article explaining the same concepts : https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

Basically, take an input image of the correct size, then apply a series of random filters to it. Now use max pooling to further reduce the images and highlight the “features” that they depict. After doing this several times, you end up with a set of features. These are then connected to a perceptron which gets trained to associate different weights to the various features to classify the input image.

Here is another great series of articles explaining ML to artists : https://ml4a.github.io/ml4a/how_neural_networks_are_trained/

********

Kits for art/design + tech workshops / open source projects I could make :

Take project I’ve done / not completed and turn it into a kit ?