Engendro 3D™ on Github

Finally, Engendro 3D, the Python based OpenGL game engine arrives to Github. If you want to try the engine so far, to criticize, help or  whatever, visit

https://github.com/jr-garcia/Engendro3D

to clone or fork and try it.

While the published version is still a very early and incomplete alpha, I needed to upload the work I have been doing on this engine and the supporting projects, at least as a backup… and I’m very happy for  doing it, since just yesterday my disk failed, taking with it a lot of work. But now I have an almost updated copy!

I’m not a friend of backups, and this have slowed down the progress on many little things I have started (and never finished) in the past, including the engine, when I sufer a failure, but now I can just clone from the repos 😀

Engendro 3D Progress Report (1 year, 7 months)

This little engine that is not born yet is still developing well. Right now, there are some news to share with anybody who  wants to hear:

  • GPU Animations are up and running.
  • I improved the performance by 10x (specially good for animated meshes) after converting two lines of the uniform upload code to Cython
  • When adding a ‘fullscreen effects’ framework, I placed the code in the wrong place and it ended up being capable of overriding the entire rendering path, so I implemented an easy-to-override rendering pipeline without even planning to (I’ll use it to add deferred rendering one day).

Sometimes the progress feels very slow on  this, but then it speeds up. I hope it will be faster now.

And next, the funny pictures:

sun

Dwarf model downloaded from animium.com, but originally belongs to http://www.psionicgames.com

Small blog edition

After more than a year with this little blog I have noticed that almost all the visits are going to the entry SharpDX multiple material performance and effects.  While there is no problem with that narrow field of interest on the other topics of the blog, most people are reaching that post by the wrong search terms, so the ‘article’ will just disappoint them (visitors often expect high level stuff related to shaders wich the article has none). Since I do not want people wasting time on stuff they don’t care (and since no one cares about stories on rotating ducks and such) I have decided to edit the entries to remove useless parts and from now on, everything I post will have the only thing all of us care about: code snippets.

Or else I will (try to) avoid publishing it.

Distance fields for font rendering (Part 2)

To properly test the font rendering from distance fields,  I adapted chunks of code from Here and Here, to make a Python signed distance field creator:

from PIL import Image
import os
from subprocess import call
from time import time
from math import sqrt


def check(limit, ox, oy, px, py, current):
    lp = getpixel(px, py)
    if lp == -1:
        return current
    if lp != limit:
        nw = float(ox - px)
        nh = float(oy - py)
        new = (nw * nw) + (nh * nh)
        if new < current:
            current = new
    return current


def getField(limit, searchLen):
    print('Calculating field {0}...'.format(str(limit + 1)))
    if limit == 1:
        limit = 255
    distances = []
    for w in range(oim.size[0]):
        distances.append([])
        for h in range(oim.size[1]):
            current = limit if limit > 0 else (searchLen * searchLen)
            for x in range(w - searchLen, w + searchLen):
                for y in range(h - searchLen, h + searchLen):
                    current = check(limit, w, h, x, y, current)
            current = int(sqrt(current))
            current *= int(g)
            distances[w].append(current)
    return distances


def getpixel(x, y):
    try:
        p = inpixels[x, y][0]
        return p
    except IndexError:
        return -1


inPath = os.path.dirname(__file__)
outPath = os.path.join(inPath, 'out.png')
inPath = os.path.join(inPath, 'in.png')
oim = Image.open(inPath).convert('RGBA')
img = Image.new('RGBA', oim.size, (0, 0, 0, 0))

ct = time()

inpixels = oim.load()
outpixels = img.load()

search = 8
last = 0.0
g = 255 / search

field1 = getField(0, search)
ct = time() - ct
print('Took: {0}'.format(str(round(ct, 1))))
ct = time()
field2 = getField(1, search)
ct = time() - ct
print('Took: {0}'.format(str(round(ct, 1))))
ct = time()
print('Mixing fields...')

for cw in range(oim.size[0]):
    for ch in range(oim.size[1]):
        fc1 = 255 - field1[cw][ch]
        fc2 = field2[cw][ch]
        fc = int((fc1 + fc2)/2)
        outpixels[cw, ch] = (fc, fc, fc, 255)

ct = time() - ct
print('Took: {0}'.format(str(round(ct, 1))))
ct = time()

print('Resizing and saving output image (128 width)...')
rf = float(oim.size[0]) / 128
img = img.resize((128, int(oim.size[1] / rf)), Image.BILINEAR)

img.save(outPath)

ct = time() - ct
print('Took: {0}'.format(str(round(ct, 1))))

if os.name.startswith('darwin'):
    call(('open', outPath))
elif os.name == 'nt':
    os.startfile(outPath)
elif os.name == 'posix':
    call(('xdg-open', outPath))

(Since this is only a test, the input and output file paths must be changed manually as needed).

The resulting image is certainly better suited for the font rendering:

SDF:

Signed Distance Field Test

Result (same shader shown in previous part):

SDF font render

Zoomed in (no weird artifacts like with the faked DF):

SDF font render zoomed

Unfortunately, the creation time for the whole DF is around 2 minutes for small images (256*256), and that is too much for the usage that I’m thinking. Now,  that usage involves the GPU so, could be faster to do this with the graphics card? Isn’t the purpose of the GPU to do per-pixel work anyway?

 

In the next part I will publish a complete tool to calculate the distance fields in the GPU, hopefully, in real time .

Edit:

For Python 3 compatibility, data passed to ‘outpixels’ needs to be send as int. Code updated to reflect it (and improved file paths).

Distance fields for font rendering (Part 1)

Pushing myself forward again, I will publish some entries about implementing font rendering into a texture atlas, encoded with distance fields and with support for Unicode chars (the possibility to reduce the texture size will probably allow large amounts of glyphs in one single texture) to finally show proper labels and text boxes in Engendro3D.


 

By checking info about how to render fonts in OpenGL, I found this question, which lead me to the Valve’s paper (.pdf).

The results, along with the need to have huge amounts of chars pre-rendered to a font atlas for certain languages, got my interest.

Since the Distance fields looks slightly like blurred images, I made a quick png texture in Gimp containing 6 letters (with Eufm10 font) and I used the Gaussian Blur filter (10 radius) to produce this faked distance Field:

wtx_blured

Then, with a very simple GLSL fragment shader, this is the result:

dfblured

No Bilinear interpolation:

gblured_noint

Zoomed into the ‘B’:

zoomblur

No Bilinear interpolation:

zoomblur_noint

Has outline and glow and, while the result is not the best, the logic works.

The shader I wrote can probably be improved a lot, but I will use the same for all the tests I’ll do:

#version 110

uniform float time;
uniform sampler2D tex;

void main()
{
float color = texture2D(tex,gl_TexCoord[0].st).r;
	if (color <= 0.5)
		gl_FragColor = vec4(0,0,1,1);
	else
		if (color <= 0.6){
			gl_FragColor = vec4(1,1,1,1);}
		else
		{
			if (color < 1.0)
				gl_FragColor = max(0.0,1.0 - color) * 
				vec4(.9,.4,.2,1) * 2.0 * max(sin(-time),sin(time));
			else
				gl_FragColor = vec4(0.0);
		}
}

 

In the next part I will show a Distance Field generator in Python and Pillow (PIL).

GUIs quirks and quacks

 

In the search of a ready-to-use GUI system, I came across several options, but in the end I have decided to make my own, simply because none of the options was completely fittable to my needs, specifically for the next reasons:

Maybe one of the most famous options. Looks very good and has official Python bindings.
I never managed to make it work in either Linux nor Windows. In Linux, after four hours of compiling, and a previously failed process due to missing dependencies, I gave up.
In Windows, the compiling took only two hours, only after I fixed some errors thanks to this instructions:

http://knightforged.com/cegui/

After finishing, I realized that the python bindings were not selected, so I started again, just to found severals errors raised for the lack of Boost. Unfortunately, I neither managed to compile the Python bindings for Boost.

 

Is recommended, looks promising and it appears to be usable from Python, but… It needs boost, so no further research on that.

Maybe not very well known like the others, but they are completely Python based, so no need for Boost. I tried to convert both to plain PyOpenGL, since SimplUI uses Pyglet’s event system (very different to mine) and BGEUI is based on Blender Game Engine and Fixed function (wich I won’t use), but after some days I realized that the effort needed for a conversion-integration into my engine would end up taking too much time. Maybe near to the time needed to make my own. So I gave up with pre-made solutions.

 

Result, after three days of work, the first screenshot of the first ‘Panel control’ of my GUI:

gui

Right now, the widgets support opacity and are stretched automatically. In the picture seen over a 3D duck with specular reflection. I have already one directional light. Current cost of the panel: 1ms

It only uses four vertices in a VBO and an index buffer. Then 3 unifforms for each control’s position/size, color and background image. This setup establish the possibility to render the GUI controls either as normal 2D ‘widgets’ or as 3D objects attached to the scene models and, with a little more work, controls could be rendered using instancing where available (all the GUI in one draw call).

and the gestation continues.

Python SDL2 TTF test

For cross platform windowing, input, font rendering and probably the entire 2d gui render of the engine, I have decided to use PySDL2, since I had some troubles with SFML (crashes in debug mode in PyCharm).

Next is an almost direct conversion from C to Python of the example found here:

http://www.willusher.io/sdl2%20tutorials/2013/12/18/lesson-6-true-type-fonts-with-sdl_ttf/

The Glametrix font I used comes from here:

http://www.glukfonts.pl/font.php?font=Glametrix

I placed the SDL2 and SDL2_ttf libraries in the script’s folder, inside another folder called ‘libs’ and the font in a folder called ‘font’.

Result:

sdlTTF

Code:

import os
from sys import exit
from ctypes import c_long, pointer

sdlpath = os.path.join(os.path.dirname(__file__), 'libs')
os.environ['PYSDL2_DLL_PATH'] = sdlpath

from sdl2 import *
from sdl2.sdlttf import *

def renderTexture(tex, ren, x, y):
    """
    :type ren: SDL_Renderer
    :type tex: SDL_Texture
    """

    #Setup the destination rectangle to be at the position we want
    dst = SDL_Rect(x, y)
    w = pointer(c_long(0))
    h = pointer(c_long(0))
    #Query the texture to get its width and height to use
    SDL_QueryTexture(tex, None, None, w, h)
    dst.w = w.contents.value
    dst.h = h.contents.value
    SDL_RenderCopy(ren, tex, None, dst)

def renderText(message, fontFile, color, fontSize, renderer):
    """

    :rtype : SDL_Texture
    """
    # Open the font
    SDL_ClearError()
    font = TTF_OpenFont(fontFile, fontSize)
    p = SDL_GetError()
    if font is None or not p == '':
        print("TTF_OpenFont error: " + p)
        return None

    #We need to first render to a surface as that's what TTF_RenderText
    #returns, then load that surface into a texture
    surf = TTF_RenderText_Blended(font, message, color)

    if surf is None:
        TTF_CloseFont(font)
        print("TTF_RenderText")
        return None

    texture = SDL_CreateTextureFromSurface(renderer, surf)
    if texture is None:
        print("CreateTexture")

    #Clean up the surface and font
    SDL_FreeSurface(surf)
    TTF_CloseFont(font)
    return texture

SDL_Init(SDL_INIT_VIDEO)
#Create an application window with the following settings:
window = SDL_CreateWindow(
    "SDL2 TTF test",  # window title
    SDL_WINDOWPOS_CENTERED,  # initial x position
    SDL_WINDOWPOS_CENTERED,  # initial y position
    640,  # width, in pixels
    480,  # height, in pixels
    SDL_WINDOW_RESIZABLE  # flags
)

renderer = SDL_CreateRenderer(window, -1, SDL_RENDERER_ACCELERATED)

tfi = TTF_Init()
if tfi != 0:
    print("TTF_Init")
    exit(1)

#We'll render the string "TTF fonts are cool!" in white
#Color is in RGB format
color = SDL_Color(255, 255, 255)
fontpath = os.path.join(os.path.dirname(__file__), 'font', 'Glametrix.otf')
image = renderText("TTF fonts are cool!", fontpath,
                   color, 64, renderer)

if image is None:
    exit(1)

#Getting the window size.
SCREEN_WIDTH = pointer(c_long(0))
SCREEN_HEIGHT = pointer(c_long(0))
SDL_GetWindowSize(window, SCREEN_WIDTH, SCREEN_HEIGHT)

#Get the texture w/h so we can center it in the screen
iW = pointer(c_long(0))
iH = pointer(c_long(0))
SDL_QueryTexture(image, None, None, iW, iH)
x = SCREEN_WIDTH.contents.value / 2 - iW.contents.value / 2
y = SCREEN_HEIGHT.contents.value / 2 - iH.contents.value / 2

r = 1
event = SDL_Event()
while r:
    if SDL_PollEvent(event):
        if event.type == SDL_QUIT:
            r = 0
        elif event.type == SDL_WINDOWEVENT:
            if event.window.event == SDL_WINDOWEVENT_RESIZED:
                SDL_GetWindowSize(window, SCREEN_WIDTH, SCREEN_HEIGHT)
                x = SCREEN_WIDTH.contents.value / 2 - iW.contents.value / 2
                y = SCREEN_HEIGHT.contents.value / 2 - iH.contents.value / 2
        if r:
            SDL_RenderClear(renderer)
            #We can draw our message as we do any other texture, since it's been
            #rendered to a texture
            renderTexture(image, renderer, x, y)
            SDL_RenderPresent(renderer)

SDL_DestroyTexture(image)
SDL_DestroyRenderer(renderer)
SDL_DestroyWindow(window)
SDL_Quit()

I moved the drawing stuff to the event processing so the text is drawn only if there is a change (window re-paint, resize).

PS: Somebody knows a way to avoid using ctypes pointers in the few PySDL2 functions that need them?

The march to the Penguin [Switching to Linux]

[Edit: removed stuff to go straight to the point]

My history with Linux has never been of success, because I had never been able to run a single ‘distro’ in none of my computers (all of them gone now, except the last [the Big] one), until very recently:

  • On Pentium 1 – Fail (none ran at all)
  • On Celeron D – Fail (only one entered to the desktop and froze there)
  • On Core2Duo – Fail (only Ubuntu entered to the desktop, and froze there)

Unfortunately I don’t remember the names of the versions I tried in the past, except Ubuntu. Anyway, this time I had a new and shiny graphics card that should at least solve the error I got the last time about my card not being capable of handling Unity desktop effects (on Ubuntu 10).

So, what requirements should I consider when searching for a Linux version? At the end it was very simple:

  1. Decent performance and compatibility in my two computers (yes, I’m poor but I managed to have two. Isn’t it great?!)
  2. Plenty of ‘apps’ in the repository (I don’t want to spend time compiling)
  3. Ease of use for a ‘noob’
  4. Modern look in the desktop (skinned, non flat nor squared controls, effects)
  5. 32 bits version (for the laptop)

Since the beginning I knew that one distro was already ahead of the others, simple because it is backed by a big company: Ubuntu, by Canonical.

But I am trying to escape from certain things that this company represents very well (and I didn’t liked Ubuntu desktop last time), so I started testing distributions. Specifically, this list made the testing process very easy and well informed:

http://www.techradar.com/news/software/operating-systems/best-linux-distro-five-we-recommend-1090058

Results in order:

DSL: Too much simple look. Hard to use for newcomers. Fast. Only works in the laptop.

Slax: Great. The main reason I kept trying. I would keep it, but only runs on the laptop.

Chakra: Discarded. Only 64 bits (But really wanted to try it).

Debian: It never installed under Virtualbox, so I didn’t tried it more.

Mageia: Great feeling. But runs to slow in the laptop, and the available software is too few.

Manjaro: The second best. Gives me a great feeling and runs very fast in the Big one. Won’t run in the laptop (froze in the desktop).

Sabayon: Decent speed in the laptop. Does not run in the Big one (again, froze in the desktop).

Kubuntu: Since I liked the K-desktop, I tried this Ubuntu version. It ended up being perfect for me, and runs in both computers. With the Debian-like repositories available. But is from Canonical!

OpenSuse: Never finished downloading.

Kanotix | Knoppix: Before those ended downloading, I decided to stop testing.

I’m sure that the problems I had with the other Linux versions can be fixed, but I just need something to start working.  So now I’m Using Kubuntu. And even with the big list of software available for it, I have ended compiling stuff anyway, but it probed to be easy enough, yet, not fast.at all.

compiling oxygenKubuntu_oxygen

(Me compiling Oxygen Transparent, the first thing I compiled under my first Linux installation)

PyOpenGL performance 2 and optimizations

Premature optimization is the root of all evil

Is it?

Note: frame times at the end.

The conversion of all modules from SharpDX to PyOpenGL is finished, so I ran the test program again. Result: 430 FPS (previously 649). Time to optimize.

First: check with cprofile module. Most time spent is in the shaders uniforms update, then in the events raising.

  • After reducing some gpu calls, >> +20 FPS
  • Converting many For loops to list comprehensions and map() >> +10FPS

I was worried because the achievements in frames where small, then I remembered the events reporting.

  • After removing all the onRenderEnd calls, where I was doing rotation of the dragon and window title update >> +1000 FPS

The engine ended up running at 1500 FPS (max). Amazing, but unrealistic, since casting events is completely necessary. Maybe a threaded approach will give improvement in this point.

But since frame times is the right way to measure, I converted all numbers, getting:

  • First gain: 0.1ms
  • Second: 0.04ms
  • Last: 1.5ms

Was it right? After trying the same program in my slow laptop, the timings probed to be correct… and meaningless. All of them:
The result in my laptop was of 58FPS, when before the improvements was of 50FPS. Only 2.7ms Total, after all the stuff I did and removed.

Final result: I’m done with optimizations. Maybe, optimizing before having something good is not good. I will move on to the next step, that will not be an editor this time.