Semiotics

Graphics That Instruct

No matter what kind of app a programmer wishes to make, setting off on a software project feels sluggish immediately. Drafting even a simple program feels like moving molasses as soon as the IDE is opened. It's not like other areas of using the computer, like video editing, where the difficulty arises later in the process.

Editing video, a photo, or a digital drawing is a more natural process in the beginning stages. In programming, though, writing code is always rough and unwieldy until a certain amount of momentum is achieved. It can be very painful at the outset.

A first way to address this is to make a new form of computer code that is semiotic. That is, it will be made of signs or symbols. It can be called semiotics-based programming, with the code being called semiotical code.

Clusters of instructional images, serving to represent code's actions and programmatic relationships, can dictate what the computer does in a way that is easy to jump into from the start and powerful enough to make broad, programmatic moves that typewritten computer code could never produce. The result will be programming schemes that are unlike any available to the software engineer today.

How do you tell the computer what to do with graphics and symbols? There are multiple approaches, and the first is to provide it instructional diagrams that resemble an airline safety card. Everyone can understand an airline safety card, and most people could make one if provided template actions, objects, and images in a computer app. But in this case it would be the computer that processes the instructional card.

In the example below, there is a sequence of frames containing graphical symbols. The black arrows indicate pulling, and they interact with a manipulatable object, the camera strap.

If this real-world chore were ever implemented as a computer program, it would be much easier to describe the process graphically and symbolically, like it is shown above, rather than typing out commands or making a control-flow schematic (e.g. nodes). At all times, the programming environment used by the programmer should match the overall form of the job.

As we know, plain text code isn't suitable for all circumstances, and for the regular computer user there are many user-friendly alternatives to make up for this. For creating and maintaining complex databases, for example, the typical consumer can make use of FileMaker Pro as long as he or she can use a word processor or spreadsheet program.

The node-based programming environment Grasshopper is often used by architects and product designers— people who are not professionals in computing— in order to rapidly preview algorithmically-generated 3D designs. Grasshopper enabled construction of this polyhedron structure using only nodes.

The issue for designers is that those nodes are not easy to decipher at a glance, nor do they give a pictorial illustration of what they produce at each stage. A control-flow programming environment is incongruous with what a designer would seek out to make a 3D model, in a big way, as it is more akin to electronics logic.

However, if a software development toolset allowed making programmatic moves in the form of instructive symbols, such as the familiar scissors cutting along a line, this would provide a natural approach to the coding situation. Symbols with graphics can represent an immense range of operations, with cutting along a construction line being only one. There can be a symbol that joins objects together, another that rotates and replicates an object in steps, another that traverses the environment in a certain way, and so forth. Importantly, these instructional symbols can be used in tandem, arranged to interact with one another and the objects they affect.

Product designers and architects spend time formulating instructions for building everyday objects, which makes symbol-based programming an immediate fit for them.

Arranging scenes of these graphics in sequences is what will make programming's appearance look very similar to the instructions for tightening the camera strap.

For the purpose of manipulating, generating, or traversing tree structures, for example, the following symbols could be combined with various line types (which can carry their own programmatic meanings). In other words, what the line looks like can be determined by the programmer and when it changes the programmatic meaning becomes something else. The tree-related symbols below can operate on objects provided by the programmer, just as the arrows moved the provided camera strap. The first is a snowflake, which indicates recursive branching. The second indicates a single stem that contains rows of branches. The third is a splitting action, which would indicate that making a "fork in the road" is desired.

Would these symbols just be traversing file systems, with their trees and nodes? No, because they are generic. With the above symbols it can be understood how something physical and concrete, like a geodesic dome, could be constructed semiotically inside a kind of storyboard. It can also be understood how these symbols would be multipurpose for programming, usable for both data applications (networks or trees of data) or construction of a 3D model like the geodesic dome.

Just like the camera strap instructional card shown at the top, what is animated below can be represented in a static programmatic image featuring symbols and the objects they act upon.

In a more futuristic scenario, a group of robots are shown how to put together a house. But in an immediate digital scenario, computer programs and 3D models can be put together.

(source: ziptiedomes)

You could easily construct this geodesic dome if the airline safety card format were converted for programmatic use by the computer. A sequence of steps depicted with symbols is already included in the assembly manual for this type of product. But what is seen in the geodesic dome itself is also a network, demonstrating how the same programmatic symbols would apply to a wide range of computational circumstances.

If during construction of the geodesic dome there is any material that needs to be cut, the symbol of scissors can be configured to do that in a way that is especially readable and concise. A multitude of properties are also configurable inside the symbol or its accompanying line(s).

For the professional programmer, this carries broad implications because instructional, graphical symbols can be utilized for all kinds of programming scenarios:

• operating on a grid of information, such as pixels to make an image filter.
• describing an algorithm (such as an R-tree) for it to be used as a utility.
• traversing data trees (e.g. file directories, the web's DOM) or network graphs
• defining user interaction flow (e.g. with switchback symbols)

Furniture Assembly Manuals

A starting point for explaining how a semiotics-based programming environment might initially exist is the assembly manual, commonly packaged with consumer products for the home, especially furniture.

A semiotics-based programming environment can supply templates, images, and tools for making instructional diagrams, like this furniture assembly page (from IKEA). Graphical symbols, like arrows, instruct the computer what to do with objects (in a manner just like the scissors and dotted-line symbol). In the future, the computer can be the recipient of assembly manual instructions instead of a person.

For the diagram below, alignment lines show how where to place the screws. Symbols are explaining how to join boards to make the furniture.

In the case of the earlier hexagonal sphere, the arrows above would join the rods and hubs that make up the structure. The sphere would be constructed in a manner similar to industry, with its constituent materials in plain sight, with a programming environment that offers the programmer symbols akin to those found in an assembly manual.

This is an entirely different direction for commanding a computer. With semiotics-based programming, a difference in appearance of a given symbol denotes a difference in purpose. Two of the arrows in the assembly manual image are bold to indicate pressing, while there is one that is circular, to indicate torque. Both are arrows, but the appearance that each takes determines what it does.

If you configure the traits of a dashed line, it can mean entirely different things. The conventions shown below are just for drafting, but they show what a semiotic-based programming environment can accommodate.

You can also turn such lines into a shape of any kind, set its angle if it is a line, set its fill color if any, and this will give it corresponding programmatic meaning. The conventions for that set of programmatic meanings would be established in advance, for the programming environment.

Relationships Illustrated with Semiotics

At the top of this page, we started with a sequence of images showing how to pull a camera strap into its grommet, with the arrow symbols denoting a set of actions.

But not every use of instructive symbols would involve a sequence of actions. Some symbols would be made to depict relationships.

In the diagram below, the computer is provided immediate knowledge of how to convert between units, just by setting up arrows with accompanying information.

Geodesic Dome Example

Back to the topic of action sequences, we look more into how the geodesic dome can be a case study. There are scripting language for making 3D models, but it is tedious to type out conventional plain text code, line by line, just to construct the 3D model below. (Similarly, it is quite awkward to construct and work with data trees in plain text).

An instructive, diagrammatic programming setup is more suitable for constructing a geodesic dome, and you can see how in this GIF animation, provided by an actual geodesic dome kit company, intended for its customers. Where in any programming language is there an opportunity to make a few sequential frames of symbolic instructions in order to construct this kind of object?

(source: ziptiedomes)

Although this GIF is animated, any person can envision what kinds of symbols would exist to join the rods into the hubs.

Instructional Symbols Working in Combination Will Provide Multi-layered Programming

A symbol of scissors moving along a dashed line tells you where to cut on a piece of paper.

But with the computer, you can turn this familiar symbol into something dynamic and programmatic, as we discussed. If you configure the traits of this dashed line to turn it into a shape or set its angle, this will give it programmatic meaning too. For example, what if the dashed line were double? It would serve a different function than a single dashed line.

This circle could represent a larger one that is actually 500 feet in diameter. And if you wanted to add actions taking place at a scale too small for this graphic, you could place a magnifying, popover frame next to the circle, pointing to their exact location inside the circle.

Much more can be present in a simple circle with scissors symbols (or other operations) than you might expect. Here are some graphical traits that can be assigned programmatic meanings:

• How spaced apart the dashes are.
• Whether the dash repetition is a dash, dot, arrowhead, or other graphic.
• How thick the line is.
• The terminals of the line.
• The line's color.
  • Whether the color represents the actual color of a thing or a means of color-coding the lines.
• The line's smoothness or texture.
• Whether it is a double line or a single line. (It could have more than two lines, of varying thicknesses.)
• What type of scissors symbol is used (among a set of variants that do different things).
• How many scissors are present, their orientations and sizes, and what orientation and size mean for programming.
• Other objects present.

This turns programming into something extremely expansive because even a single, straight line can contain several programmatic properties and functions at once just because of its appearance.

Furthermore, multiple lines connected can signify sequential steps. For example, let's repeat an object on a page. A dashed line representing travel (translation) moves an object a certain distance, at a certain angle. Attached to its terminal point is a deposit symbol, depositing a copy of the object onto the page. Subsequently, it is connected to a logic element that makes a decision, whether to travel in one direction or another (but this is still not a control-flow diagram). An application for this is basket weaving, where decisions are made for whether to go over or under.

But, of course, going over or under, skipping a portion of something, and other operations are relevant to many areas of computing, not just physical manufacturing and 3D models.

Here are some portrayable aspects in graphical symbols, all of which could utilize scale:

• temporal (how long to pause, how fast to move)
• spatial (how large something will grow, how far to travel, how much to scale)
• quantity (how many to generate).

Making a Programming Library Using Semiotics

A Programming Library That Converts Units of Measurement

Today's software development tools do not, in actuality, accommodate the underyling concepts that exist in common programming situations.

Take for example an essential need, making a programming library for converting units of measurement. What tools should the software development environment provide the programmer for this scenario? He could type it all out as usual with regular code, weaving all kinds of variables, objects, and functions. But that certainly is not the best way if we have a chance to upgrade our programming environment in an overarching way. What is it about this task that makes plain text code ill-suited for it?

To begin with, there are relationships here: the volume of liquid in one cup is half that of a pint, and a pint is half the volume of a quart, which is one fourth of the volume of a gallon. With units of measurement, there is a chain, spanning from smallest to largest or vice-versa. There are no such "chains" provided in today's programming languages, as there has never been an emphasis on building these kinds of concepts into the code. There are no conventions provided for defining relationships between objects.

Aren't these (mathematical) relationships between units representable with symbolic graphics? Then the programmer could deposit the names of all units on a page within boxes in an orderly way and then assign mathematical relationships (i.e. ratios) between the boxes, using established semiotic conventions.

If the computer recognized these symbols, the picture below is all that would be needed to tell the computer how to convert km to m to cm, for any situation.

Each topic of measurement (e.g. volume, mass, length) can actually be entered into the computer in terms of graphical schematics, as long as such a software development environment provides this kind of feature. It's just that none ever have, and that shows how raw current programming practices truly are. They don't anticipate what anyone will do with them; it's just functions and variables all over the place.

If you started from scratch making a set of software libraries, this would be the first feature to build into it, because it wouldn't take long to inform the computer about a wide range of relationships that exist in the real world.

Right now, making a unit conversion library is a drawn-out, laborious chore that would never be undertaken by a beginner to intermediate programmer. The problem is that it is painstaking to make this kind of programming library even though, conceptually, the topic is straightforward. The conversion library is really just units with ratios set up between them.

Using semiotic graphics, all of this can be accomplished without the unnecessary toil. The code for the semiotics-based programming library is a set of diagrams or schematics, and this is much more manageable for this situation, especially for someone new to the project who needs to maintain or modify something.

For this programming situation, all you should have to do is lay down the relationships that exist between units and have the computer take care of the rest. And it is here that we see the shortcomings in today's mindsets toward writing code. In fact, the undeveloped plain text medium should not be considered versatile; it should be consider bare.

This diagram would be all that it takes to tell the computer how to handle these units of measurement, for any situation it encounters. It shows the importance of introducing semiotics-based programming into the technology industry.

If a programmer can plot abstract relationships into a software development environment, there can be construction of programming libraries in a natural way. And because it is done in a natural way, it would move along quickly. Any such programming environment would be general purpose, with its building blocks able to accommodate all sorts of scenarios, not only for arithmetic conversions. Image compression, for example, typically involves a set of algorithm descriptions for what the computer should do to the image file in steps.

It would take a lot less time and tedium to lay out a measurement conversion library out according to a conceptual schematic, but more importantly it is a proper way to go about things. The final result on the schematic pages would be better for all involved, especially for someone new to the project who needs to maintain it. The code for the programming library is mostly a set of sheets of diagrams or schematics, and this is much more manageable.

Making a Programming Library That Carries out Graphics Transformations

It's not uncommon for a graphics programming library to sit on dozens or even hundreds of math functions, but the accepted format for programming, plain text, can only accommodate monospaced type.

Math that could be entered into the computer in a notation format (adapted to the computer) instead just exists as typewritten characters. All aspects of a math situation must be painstakingly divided into its individual functional pieces only because monospaced-type code can accommodate this alone.

A case example is an affine transform matrix. For 2D and 3D graphics, transform matrices enable operations like rotation, scale, shear, and translation. Just like the measurement library example discussed above, this type of library takes a disproportionate amount of time to assemble in today's programming environments. The math is already there and well-established. It is in fact the programming environment that cannot receive math formulas in such a way that they can be utilized for establishing a programming library.

A programmer should be able to take a set of math formulas and instruct. This is where semiotics becomes useful again. It is the semiotic graphics that will indicate what the computer should do with the formula to make a programming library.

Arrows can overlay a math equation or formula to indicate what the computer is supposed to provide the programmer in terms of utility for this programming library.

In the example of an affine matrix, the semiotic graphics, the arrows and other elements, indicate how a math equation it is to be used. Then it is the job of the computer to sort out for itself how it would like to arrange this in terms of functions, objects, and variables.

Semiotics-Based Programming for The Prosumer

Semiotical Programming Can Be a Powerful Way for The Advanced Computer User to Handle Computers

There is actually a broad spectrum of people who use computers; it is not just regular consumers on one side and then software engineers on the other. In the middle area, where most computer enthusiasts are, there are a lot of people who would like the computer to do advanced things for them, but they are not a fit for today's programming because they know it is somehow tailored to only one type of individual. They might not be able to articulate this at all times. It is the overall form of programming itself that is the problem for them, not always that these so-called "prosumer" people who find code unpleasant to deal with are lacking advanced technical interests. Many tech hobbyists who find programming inaccessible today would certainly be engaged in writing apps if the form that code takes weren't so primitive and inhospitable. Semiotics-based programming offers a path towards this outcome.

Tech Hobbyists Want Something Like Semiotics-Based Code

A large number of computer enthusiasts are addicted to building or purchasing new computers despite owning several already. They want to buy the latest machine to run benchmark tests on it, capture video of their home setup, and then post footage to YouTube. They will buy tiny computers, then large ones, and spend time coming up with frivolous uses for them, like running video game emulators. Several months to a year later, they will build another computer. Or they will find another device for running a retro video game emulator. They don't really know what to do with computers, but like to see the computers do things. In ways unrecognized, computers are limited for large portions of the population despite carrying volumes of capabilities underneath.

There is a strong case to be made that this results from the state of computer programming being very poor, not just these hobbyists carrying an addiction to computers. That's because what these prosumers are actually trying to do is assemble rigs that can open up for them a wide world of heavy computer capabilities. It's just that most of those capabilities are going to be out of their reach until computer code changes for the better. This hobbyist can sense that a great deal is untapped in every new computer that he builds, so he keeps purchasing new computer setups (the latest AMD CPU, the latest Mac Mini, the newest graphics card, etc.). However, the programming environments (e.g. Visual Studio) that he could use to do heavy tasks of his own making are not suited for him because he is actually a prosumer. To this hobbyist, an IDE is too complicated, it requires taking extra courses, or perhaps it is just unattractive.

Unsure what to do next, he starts trying out the mainstream, professional media apps that run on his new hardware. It's just that the apps he installs are exactly the same as before, and they are not any more fun to use just because the computer he assembled is more powerful than last year's. He will then do speed tests to compare how fast everything is running next to everything else on the market, past and present.

A dilemma exists for the prosumer that keeps this cycle in place: when today's computers are simple to program, they cannot do much beyond scripting, which is too limited and dull for such a feature-rich machine. And web pages can also be rather tedious and superficial. But, when a computer gets difficult to program, which is what is needed to make substantial apps, the software development tools are unpleasant or inaccessible to this person, the computer enthusiast. Not every prosumer who likes to use computers as a hobby wants to write code. But he would if code existed differently.

When programming with semiotics becomes widely available, the computing world will undergo a shift. Code made of graphical symbols will be as accessible to the prosumer as a scripting language but as powerful as the plain text code that career programmers now use for apps, because that code has never been very effectual, especially for today's needs. Making code out of symbolic graphics is a higher level of using the computer that translates to power at the same time. It will not result in a reduction of provided functionality, nor will it be overly simplified. Semiotics-based programming has likelihood of changing the entire landscape of the computing industry, IT, and video games.

It isn't that the professional and prosumer will be merged, though. When semiotics-based programming languages are available, the professional will be doing the more advanced or specialized tasks within a semiotics-based software project, while the hobbyist will be doing the tasks at his level that were earlier beyond his skill set. New types of projects will be available to the professional programmer of today, while tasks that are now only the domain of software engineers will be something a hobbyist can do without asking for help.

On-Camera Hand Gestures Recognized by Machine Learning

Hand Gestures in VR

The latest VR headset from Oculus, Oculus Quest 2, supports one hand gesture, the system menu hand gesture.