【技術解説】Digital pottery with Azure Kinect & pix2pix

Hi, I am an interaction engineer/designer MAO (@rainage) of The Designium.

In this article, I would introduce an interesting project, "Digital pottery".

Introduction

The digital pottery project is an R&D prototype during I tried the Azure Kinect.

I want to make a demo to make shape with body movement for practice how to control skeleton joints of Azure Kinect.

The number of skeleton joints of Azure Kinect is more than Kinect v2, and the tracking quality is really good after the body tracking SDK v0.95.

Make pottery shape with skeleton joints

Main steps:
(1) select enabled skeleton joints
In this demo, I only select the main joints without eyes, nose, ears, and fingers.

(2) pick the points great than the threshold setting.
for example, I set the threshold as joint x position great than 0f.

(3) make lofting line
order picked points with y value, then adding start & endpoints for lofting shape line.

private void SortLinePoints(ref List<Vector3> ptList) {
   if (ptList == null)
      return;
   if (ptList.Count <= 2)
      return;

   List<Vector3> _tmpList = new List<Vector3>();

   // select threshold : only using the point on left screen
   for(int i=1; i<ptList.Count; i++) {
      if(ptList[i].x>=0f) {
         _tmpList.Add(new Vector3(ptList[i].x, ptList[i].y, 0f));
      }
   }

   // make sure the line at least 2 points
   if (_tmpList.Count < 2) {
      _tmpList.Clear();
      _tmpList.Add(new Vector3(0.01f, -1f, 0f));
      _tmpList.Add(new Vector3(0.01f, 0f, 0f));
   }

   // order points by y axis
   ptList.Clear();
   ptList = _tmpList.OrderBy(obj => obj.y).ToList();

   // add 2 points as stat & end for better shape
   ptList.Insert(0, new Vector3(0f, ptList[0].y, 0f));
   ptList.Add(new Vector3(0f, ptList[ptList.Count - 1].y, 0f));
   _tmpList = null;
}

(4) make 3D loft shape
For using the points as lofting lines to make loft shape, I needed a lightweight and real time modeling tool for lofting shape.  I found the "CTModeler" package, it is nice for making some simple geometry shape.

Kronopath/CTModeler

using CT;
...

void SetDefualtPoints() {
   _points.Clear();
   _points.Add(new Vector3(.0f, .0f, .0f));
   _points.Add(new Vector3(.0f, .1f, .0f));
}

void MakeLoftMesh() {
   if (_points == null)
       return;
   // for no-user tracking
   if (_points.Count < 3)
       SetDefualtPoints();
   cupGenerate.SetPoints(_points);
   cupGenerate.RegenerateMesh();
}

Pix2pix

After generating the loft shape, I added an interesting effect, called pix2pix (image-to-image translation with a deep neural network), on the resulting image.

I used the kejiro's pix2pix on Unity example, there is a post processing example of this.

keijiro/Pix2Pix

As you can see, using the example default setting will be like a cat cup. XD

I replace the pre-trained model (copy it into Assets/StreamingAssets) and fixed the post-processing setting. The resulting effect will be more like ink-painting.

The Designium.inc
・Interactive website
・Twitter
・Facebook

Editorial Note  編集後記

こんにちは、広報のマリコです！陶器といえば、私はルーシー・リーの器や波佐見焼なども大好きなのですが、実際につくったことはありません。修学旅行で絵付けをしたことがあったかな？くらいなので、今回のMaoのコンテンツは身体をつかって陶器をつくれるのは楽しそうだなと。しかも最後の作品なんかは絵として飾ったらちょっとかっこ良さそうですね✨