m (→color%20) |
m (→Video) |
||
(39 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
[[File:ImageCompute-header.png|700px]] | |||
=color%20= | =color%20= | ||
==Context== | ==Context== | ||
Line 6: | Line 7: | ||
==Concept== | ==Concept== | ||
Digital images can be treated as pseudo-random distributions of color. If these are interpreted as three-dimensional point-clouds, it is possible to overcome the planar image by adding depth and creating new ways to interpret the image by exploring the color distribution in its purest form. After using different color metrics to calculate the three-dimensional position of every point, complex structures and patterns emerge, colors shift, strains form and clusters of colors move from position to position. | |||
---- | ---- | ||
==Approach== | ==Approach== | ||
For every pixel in the image a colored voxel is spawned. The position of the voxel is than determined by different mapping functions that map the pixel (x, y, r, g, b) to position in space (x, y, z). In the default mapping every voxel is on a plane (z = 0) and the position in the plane is directly determined by the position in the original two-dimensional image. | For every pixel in the image a colored voxel is spawned. The position of the voxel is than determined by different mapping functions that map the pixel (x, y, r, g, b) to position in space (x, y, z). In the default mapping every voxel is on a plane (z = 0) and the position in the plane is directly determined by the position in the original two-dimensional image. | ||
[[File:ImageCloud-Default-Mapping.png|Default mapping| | [[File:ImageCloud-Default-Mapping.png|Default mapping|360px]] | ||
[[File:ImageCloud-Simple2.gif|RGB Space|360px]] | |||
Since the virtual space is three-dimensional space, the third dimension or depth (z-axis in this case) can be used to convey additional information thus revealing a new perspective on the image. The red channel could for example be mapped to the z-coordinate of every voxel. But we aren't limited to only using the z-axis. Since every digital image represents a subset of the rgb-space (cube), we can also map every pixel to a position in the rgb-space (represented as a cube). | Since the virtual space is three-dimensional space, the third dimension or depth (z-axis in this case) can be used to convey additional information thus revealing a new perspective on the image. The red channel could for example be mapped to the z-coordinate of every voxel. But we aren't limited to only using the z-axis. Since every digital image represents a subset of the rgb-space (cube), we can also map every pixel to a position in the rgb-space (represented as a cube). | ||
Line 18: | Line 20: | ||
File:ImageCloud-Approach2.png|Red Color Channel Depth | File:ImageCloud-Approach2.png|Red Color Channel Depth | ||
File:ImageCloud-Approach3.png|RGB Color Space | File:ImageCloud-Approach3.png|RGB Color Space | ||
File:ImageCloud-Approach4.png|HSV Color Space | |||
</gallery> | </gallery> | ||
===Mapping=== | ===Mapping=== | ||
Since there are more three-dimensional color-representations of digital color such as HSV (or HSL) and YCbCr that have their own three-dimensional interpretations (HSV a cylinder, YCbCr a cube) and every singel channel can be mapped to the z-axis the following mappings were implemented | Since there are more three-dimensional color-representations of digital color such as HSV (or HSL) and YCbCr that have their own three-dimensional interpretations (HSV a cylinder, YCbCr a cube) and every singel channel can be mapped to the z-axis the following mappings were implemented: | ||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
Line 68: | Line 71: | ||
| Saturation | | Saturation | ||
| Value | | Value | ||
|- | |||
| Y Channel Depth | |||
| Original | |||
| Original | |||
| Y (luma component) | |||
|- | |||
| Cb Channel Depth | |||
| Original | |||
| Original | |||
| Cb | |||
|- | |||
| Cr Channel Depth | |||
| Original | |||
| Original | |||
| Cr | |||
|- | |||
| YCbCr Space | |||
| Y (luma component) | |||
| Cb | |||
| Cr | |||
|} | |} | ||
===Implementation=== | |||
Depending on the resolution of the original image the amount of voxels that have to be rendered and updated can easily exceed the millions. Since the resolution of VR-headsets is still limited aroud 2k it is sufficient to rescale the input-image to about 1024*1024 = 1048576 pixels. This is still too much to render and update every voxel as an independent object in real-time. A solution to this is using instancing on the GPU and just passing the color and position of the instances as a buffer. When using direct instanced drawing in Unity the calculation of the bounds of the object is very important to support native-features like view-frustum-culling. | |||
To calculate the position- and color-buffer in realtime a compute-shader is used that can easily compute millions of independent calculations in parallel per frame. Compute-Shader inputs can be changed at runtime enabling transitions between diffrent resolutions, mappings and positions. | |||
===Transitions=== | ===Transitions=== | ||
[[File:ImageCloud-Transition1.mp4| | Because the compute-shader can run every frame transitions between different states can be calculated. In this case a simple particle-based system was used where every particle (voxel) has a position, velocity and acceleration. | ||
[[File:ImageCloud-Transition2.mp4| | |||
[[File:ImageCloud-Transition3.mp4| | [[File:ImageCloud-Transition1.mp4|Mapping Transition|245px]] | ||
[[File:ImageCloud-Transition2.mp4|Image Transition|245px]] | |||
[[File:ImageCloud-Transition3.mp4|Resolution Transition|245px]] | |||
Exmaples of transitions between mappings (left), images (middle) and resolutions (right). | |||
{{#ev:youtube|cA7D-l4XI84|735}} | |||
===Manipulation=== | ===Manipulation=== | ||
Since VR-headsets were rare this semester I tried to use the [https://www.ultraleap.com/product/leap-motion-controller/ Leap-controller] to capture the movement of the hands and try to manipulate the image with it. | |||
While I was able to set up and calibrate the controller in Unity, the captured motion was very inconsistent and jittery and the movement of the hand was limited. Since the camera had to be static the added value of the three-dimensionality was limited, using a VR-headset in combination with the Leap-controller might solve this problem. | |||
[[File:ImageCloud-Manipulation1.mp4|Hand Manipulation|735px]] | |||
Another problem with manipulating the voxels is that the particle representations of the voxels are not part of the physics-world and cannot collide with the other objects. Every collision of every particle with every object has to be calculated in the compute-shader. While I was able to implement simple interaction with the center of one hand, I failed to implement a working collision for simple geometric forms in the compute-shader that was efficient enough to run every frame. This is possible, but GPU-collision is a difficult task that takes a lot more work than expected. | |||
---- | ---- | ||
== | ===Video=== | ||
The uniforms of the compute-shader can be updated during runtime. With the use of the Unity-Video-Player the input image can be the current frame of a video. If the video is updated every frame the particles get a new target position every frame creating a moving cloud of particles that create ever-emerging patterns of movment. | |||
---- | {{#ev:youtube|ZSVnI2nV1V4|735}} | ||
==Media== | ==Media== | ||
[[File:Creating a Graph 11.05.2021 08_56_28.png|360px]] | |||
[[File:Creating a Graph 11.05.2021 08_57_26.png|360px]] | |||
[[File:Creating a Graph 11.05.2021 08_58_03.png|360px]] | |||
[[File:Creating a Graph 11.05.2021 08_59_21.png|360px]] | |||
[[File:Creating a Graph 11.05.2021 09_00_23 (1).png|360px]] | |||
[[File:Creating a Graph 11.05.2021 09_00_44.png|360px]] | |||
[[File:Creating a Graph 11.05.2021 09_01_11.png|360px]] | |||
[[File:Creating a Graph 12.05.2021 14_03_36.png|360px]] |
Latest revision as of 05:39, 13 May 2021
color%20
Context
In this project I wanted to revisit the idea of treating digital images as spatial pseudo-random distributions of color-entities (pixels in 2d, voxels in 3d). As digital color is mostly stored as a triplett of color-channels (red, green and blue), it can easily be mapped into a three-dimensional space. We can further take advantage of the possible perception of true three-dimensionality using a VR-Headset. Having the image represented as a cloud of voxels in digital environment, we can then use the VR-controllers to manipulate the image in this three-dimensional domain using our hands by pushing and pulling parts of the image-cloud and remapping the cloud to the two-dimensional canvas. This technique might provide a more intuitive approach of manipulating images and their colors.
Concept
Digital images can be treated as pseudo-random distributions of color. If these are interpreted as three-dimensional point-clouds, it is possible to overcome the planar image by adding depth and creating new ways to interpret the image by exploring the color distribution in its purest form. After using different color metrics to calculate the three-dimensional position of every point, complex structures and patterns emerge, colors shift, strains form and clusters of colors move from position to position.
Approach
For every pixel in the image a colored voxel is spawned. The position of the voxel is than determined by different mapping functions that map the pixel (x, y, r, g, b) to position in space (x, y, z). In the default mapping every voxel is on a plane (z = 0) and the position in the plane is directly determined by the position in the original two-dimensional image.
Since the virtual space is three-dimensional space, the third dimension or depth (z-axis in this case) can be used to convey additional information thus revealing a new perspective on the image. The red channel could for example be mapped to the z-coordinate of every voxel. But we aren't limited to only using the z-axis. Since every digital image represents a subset of the rgb-space (cube), we can also map every pixel to a position in the rgb-space (represented as a cube).
Mapping
Since there are more three-dimensional color-representations of digital color such as HSV (or HSL) and YCbCr that have their own three-dimensional interpretations (HSV a cylinder, YCbCr a cube) and every singel channel can be mapped to the z-axis the following mappings were implemented:
Name | X-Axis | Y-Axis | Z-Axis |
---|---|---|---|
Red Color Channel Depth | Original | Original | Red |
Green Color Channel Depth | Original | Original | Green |
Blue Color Channel Depth | Original | Original | Blue |
RGB Space | Red | Green | Blue |
Hue Channel Depth | Original | Original | Hue |
Saturation Channel Depth | Original | Original | Saturation |
Value Channel Depth | Original | Original | Value |
HSV Space (Cylinder) | Hue | Saturation | Value |
Y Channel Depth | Original | Original | Y (luma component) |
Cb Channel Depth | Original | Original | Cb |
Cr Channel Depth | Original | Original | Cr |
YCbCr Space | Y (luma component) | Cb | Cr |
Implementation
Depending on the resolution of the original image the amount of voxels that have to be rendered and updated can easily exceed the millions. Since the resolution of VR-headsets is still limited aroud 2k it is sufficient to rescale the input-image to about 1024*1024 = 1048576 pixels. This is still too much to render and update every voxel as an independent object in real-time. A solution to this is using instancing on the GPU and just passing the color and position of the instances as a buffer. When using direct instanced drawing in Unity the calculation of the bounds of the object is very important to support native-features like view-frustum-culling. To calculate the position- and color-buffer in realtime a compute-shader is used that can easily compute millions of independent calculations in parallel per frame. Compute-Shader inputs can be changed at runtime enabling transitions between diffrent resolutions, mappings and positions.
Transitions
Because the compute-shader can run every frame transitions between different states can be calculated. In this case a simple particle-based system was used where every particle (voxel) has a position, velocity and acceleration.
Exmaples of transitions between mappings (left), images (middle) and resolutions (right).
Manipulation
Since VR-headsets were rare this semester I tried to use the Leap-controller to capture the movement of the hands and try to manipulate the image with it. While I was able to set up and calibrate the controller in Unity, the captured motion was very inconsistent and jittery and the movement of the hand was limited. Since the camera had to be static the added value of the three-dimensionality was limited, using a VR-headset in combination with the Leap-controller might solve this problem.
Another problem with manipulating the voxels is that the particle representations of the voxels are not part of the physics-world and cannot collide with the other objects. Every collision of every particle with every object has to be calculated in the compute-shader. While I was able to implement simple interaction with the center of one hand, I failed to implement a working collision for simple geometric forms in the compute-shader that was efficient enough to run every frame. This is possible, but GPU-collision is a difficult task that takes a lot more work than expected.
Video
The uniforms of the compute-shader can be updated during runtime. With the use of the Unity-Video-Player the input image can be the current frame of a video. If the video is updated every frame the particles get a new target position every frame creating a moving cloud of particles that create ever-emerging patterns of movment.