RSS

OpenIR Interrupts

March 1st, 2015 • Electronics, Software Development, STM8Comments Off

A few nights ago I was working on implementing the processing the command sequences stored in the STM8S into IR pulses. The Win32 configuration application takes a series of high / low transitions measured in microseconds and turns the infra-red LED on and off accordingly. Doing this would require a signal capable of being triggered to the nearest microsecond.

This is where the plot fell apart.

Looking at the application now I think it is a case of over engineering the problem. By modifying the design parameters to be more realistic the problem can be overcome.

How Fast Do We Need To Be?

Previous articles have shown how a 2-3 MHz pulse can be generated by setting the system clock to 16MHz and toggling a port in a while loop. It has also been shown how Timer 2 can be used to generate a square wave using an interrupt based upon the clock pulse. The challenge is to generate and interrupt at a high enough frequency.

From the start it was recognised that the generation of a 1 MHz signal was ambitious using interrupts. Raising an interrupt on the STM8S takes 9 clock cycles and working on a 16MHz clock this only leaves 5 clock cycles for the work of the Interrupt SubRoutine (ISR) before the microcontroller returns to sleep and the whole cycle restarts.

So the question becomes, is this a problem?

Breaking out the calculator reveals that a 38KHz signal (a common frequency used for infra-red signals) has a period of 26 microseconds. So if we were to send a single pulse the ISR would have to respond within 26 microseconds. Having a carrier signal would not make sense if there were less than two carrier signal pulses within the infra-red signal. This would result in a minimum frequency of 19KHz which is well within the capabilities of the STM8S.

As I said, over engineering, or more likely, worrying about a problem which does not really exist.

Recreating the Nikon Shutter Trigger Signal

The changes which have been made are all aimed at making the remote control a universal remote control. It is i=now time to get back to the orignal aims of the project, namely to be able to trigger a Nikon camera.

The last post demonstrated the ability to create a configurable sequences of pulses / pauses and store these in the EEPROM. The next step is to convert these into a pulse sequence based upon a 38KHz carrier signal.

Although the predominant carrier frequency for an infra-red remote control is 38KHz, this implementation allows for the carrier frequency to be configured. In order to do this the software will need to change the counter and prescalar values of Timer 1, Channel 4. Four bytes of the EEPROM are reserved for these two values. The Windows application performs most of the work by calculating the required values:

int count = 0;
int prescalar = 0;
bool error = false;

if (value < 1)
{
    error = true;
}
else
{
    count = (int) (CLOCK_FREQUENCY / value);
    while ((prescalar < 65536) &amp;&amp; (count > 65535))
    {
        prescalar++;
        count = (int) (CLOCK_FREQUENCY / (value * prescalar));
    }
    error = (prescalar == 65536);
}

if (error)
{
    float maxFrequency = CLOCK_FREQUENCY / 65535;
    throw new ArgumentOutOfRangeException("CarrierFrequency", string.Format("Carrier frequency should be between 1 and {0} Hz.", maxFrequency));
}
else
{
    _memory[PULSE_DURATION_OFFSET] = (byte) (count >> 8);
    _memory[PULSE_DURATION_OFFSET + 1] = (byte) (count &amp; 0xff);
    _memory[PRESCALAR_OFFSET] = (byte) (prescalar >> 8);
    _memory[PRESCALAR_OFFSET + 1] = (byte) (prescalar &amp; 0xff);
}

value is the desired frequency in Hz. The code cycles through the possible prescalar values until the right combination of counter and prescalar vales are obtained. The prescalar for Timer 1 is a value between 0 and 65535. One thing to note is that the value of the counter and the prescalar may not always return an exact frequency, for example, a carrier signal of 38KHz results in counter and prescalar values which generate a 38,004Hz signal. Not exact but given that the remote is using the internal oscillator we cannot expect an exact output anyway.

The perfectionist in me finds this to be disturbing but it is something I’ll have to live with.

Back on the remote control the application needs to use the configuration information in the EEPROM in order to regenerate the carrier signal for the remote control.

//
//  Set up Timer 1, channel 4 to output a PWM signal (the carrier signal).
//
void SetupTimer1()
{
    TIM1_ARRH = *((unsigned char *) EEPROM_PULSE_DURATION);
    TIM1_ARRL = *((unsigned char *) (EEPROM_PULSE_DURATION + 1));
    TIM1_PSCRH = *((unsigned char *) EEPROM_PRESCALAR_OFFSET);
    TIM1_PSCRL = *((unsigned char *) (EEPROM_PRESCALAR_OFFSET + 1));
    //
    //  Now configure Timer 1, channel 4.
    //
    TIM1_CCMR4_OC4M = 7;    //  Set up to use PWM mode 2.
    TIM1_CCER2_CC4E = 1;    //  Output is enabled.
    TIM1_CCER2_CC4P = 0;    //  Active is defined as high.
    //
    //  Work out the 50% duty cycle based upon the count.
    //
    unsigned short fiftyPercentDutyCycle;
    fiftyPercentDutyCycle = *((unsigned char *) EEPROM_PULSE_DURATION);
    fiftyPercentDutyCycle <<= 8;
    fiftyPercentDutyCycle += *((unsigned char *) (EEPROM_PULSE_DURATION + 1));
    fiftyPercentDutyCycle >>= 1;
    TIM1_CCR4H = ((fiftyPercentDutyCycle >> 8) &amp; 0xff);
    TIM1_CCR4L = fiftyPercentDutyCycle &amp; 0xff;
    //
    TIM1_BKR_MOE = 0;       //  Disable the main output.
    TIM1_CR1_CEN = 1;       //  Enable the timer.
}

Much of this code uses the counter values from the EEPROM to set up the PWM on Timer 1 Channel 4.

Two changes which have been made which do not impact the frequency of the carrier signal are the last two lines of the method. Previous versions of the code turned the time output on and the timer off. Here the Timer is turned on and the output off. This has been done to remove the need for the AND gate in the LED driver section of the circuit:

LED Driver With AND Gate

LED Driver With AND Gate

The AND gate takes the PWM signal from the STM8S along with an output enable signal (Port D, pin 3) from the STM8S. Changing the code to disable the timer removes the need for the output enable signal. The timer output is set to active high and when the timer is disabled the output pin of the timer goes into a high state. This problem is resolved by leaving the timer active and disabling the main output of the timer.

Another small change is to Timer 2, the timer used to determine the duration of the output of the carrier frequency. The Windows application takes the output and pause periods in microseconds. The Nikon shutter signal looks like this:

Nikon Pulse Sequence in Windows Application

Nikon Pulse Sequence in Windows Application

When this is encoded in the EEPROM the sequence data looks as follows:

Nikon IR Sequence in EEPROM

Nikon IR Sequence in EEPROM

The memory is encoded as follows:

Memory Offset Length Description
0x00 0x10 Name of the remote control.
0x10 0x02 Timer 1 counter value for the carrier signal.
0x12 0x02 Timer 1 prescalar for the carrier signal.
0x14 0x01 Number of pulse sequences / commands in the EEPROM.
0x20 0x40 Length of the sequences / commands in the sequence table.
0x60 0x1f8 Sequence / command data.

Each of the commands is encoded as follows:

Memory Offset Length Description
0x00 0x08 Name of the command.
0x08 1 Number of pulses / pauses in the command (n).
0x09 n * 2 Command / pause duration in microseconds.

Now let’s look at how this data can be retrieved and used.

Timer 2 can be configured with a clock prescalar in the range 1 to 32,768. With a clock frequency of 16MHz a prescalar of 16 would drop the clock frequency to 1MHz. Doing this makes the calculation of the counter values as simple as recording the number of microseconds in the 16-bit values used for the counters. The setup code for Timer 2 becomes:

//
//  Setup Timer 2 ready to process the pulse data.
//
void SetupTimer2()
{
    TIM2_PSCR = 4;
    TIM2_ARRH = 0;
    TIM2_ARRL = 0;
    TIM2_EGR_UG = 1;
    TIM2_IER_UIE = 1;       //  Enable the update interrupts.
}

The Timer 2 interrupt handler currently looks are follows:

//
//  Timer 2 Overflow handler.
//
#pragma vector = TIM2_OVR_UIF_vector
__interrupt void TIM2_UPD_OVF_IRQHandler(void)
{
    _currentPulse++;
    if (_currentPulse == _numberOfPulses)
    {
        //
        //  We have processed the pulse data so stop now.
        //
        PD_ODR_ODR3 = 0;
        TIM2_CR1_CEN = 0;
        TIM1_BKR_MOE = 0;
        _currentState = STATE_WAITING_FOR_USER;
    }
    else
    {
        TIM2_ARRH = *_pulseDataAddress++;
        TIM2_ARRL = *_pulseDataAddress++;
        TIM1_BKR_MOE = !TIM1_BKR_MOE;       //  Toggle the T1 output.
        TIM2_CR1_URS = 1;
        TIM2_EGR_UG = 1;
    }
    //
    //  Reset the interrupt otherwise it will fire again straight away.
    //
    TIM2_SR1_UIF = 0;
}

Where _pulseDataAddress points to the next pulse to be processed and _numberOfPulses holds the count of the number of pulses in this sequence.

The code below walks through the list of commands looking for the pulse data for the sequence number held in the command variable.

//
//  We have enough command data.  Now work out where the command is.
//
_pulseDataAddress = (unsigned char *) (EEPROM_SEQUENCE_DATA_OFFSET);
unsigned char *length = (unsigned char *) EEPROM_SEQUENCE_LENGTH_TABLE_OFFSET;
for (unsigned char index = 0; index < command; index++)
{
    _pulseDataAddress += *((unsigned char *) length);
}

Now we have the pulse sequence is is necessary to set up Timer 2 and the global variables pointing to the pulse data and the number of pulses.

//
//  Now start processing the pulse data for the specified command.
//
_currentState = STATE_RUNNING;
_pulseDataAddress += MAX_PULSE_NAME_LENGTH;     //  Skip the name.
_numberOfPulses = *_pulseDataAddress++;
_currentPulse = 0;
TIM2_ARRH = *_pulseDataAddress++;
TIM2_ARRL = *_pulseDataAddress++;

The final step is to turn on Timer 1 output and start Timer 2 running:

//
//  Now we have everything ready we need to force the Timer 2 counters to
//  reload and enable Timers 1 &amp; 2.
//
TIM2_CR1_URS = 1;
TIM2_EGR_UG = 1;
TIM1_BKR_MOE = 1;
TIM2_CR1_CEN = 1;

Calling TransmitPulseData should allow the command sequence to be output. Adding the line of code TransmitPulseData(0); after the configuration code and hooking up the logic analyser results in the following output:

Nikon IR Sequence in Logic Analyser

Nikon IR Sequence in Logic Analyser

Conclusion

OpenIR has reached the point where command sequences can be edited and stored in the EEPROM and a single command sequence can be transmitted on a configurable carrier frequency.

Next steps:

  1. The pulse sequence storage in the Windows application still needs to be tidied up.
  2. When loading the EEPROM data the Windows application needs to regenerate the data for the list box containing the list of sequences.
  3. Add commands to send a determined sequence / command number.
  4. Redesign the board to remove the AND gate from the LED output.

Once the above has been completed additional connectivity options can be investigated.

For those who are interested, the source code for STM8S running the remote control is available for download.

Editing OpenIR Commands

February 22nd, 2015 • Software Development, STM81 Comment »

The OpenIR application can now store and retrieve the contents of the EEPROM. The previous article demonstrated how the basic IR parameters can be stored and retrieved from the STM8S EEPROM. It is now time to start to store and retrieve command / pulse sequences in the EEPROM ready for the STM8S to process.

Pulse Sequences

In the January design update a template for the EEPROM layout was presented. The final part of the EEPROM was the pulse sequences data. What has become apparent is that in addition to the pulse count data we should also be storing a meaningful name for the command. The names are not required by the STM8S as it can simply take a command number but they make the commands more meaningful when viewed in the Windows application. It is for this reason that we need to store the command name in the remote control.

Pulse Sequence Length Table

The Pulse Sequence Length table contains a list of the number of bytes in each pulse sequence. Each entry gives the offset from the start of the pulse data for each of the commands the remote control can transmit to a remote device.

Pulse Data

The pulse data table contains the information about the commands themselves. The first eight bytes contain the name of the command. The STM8S will ignore this as it is not needed in order to transmit the command to the remote device. The next byte contains the number of pulses/transitions in the command. The following pairs of bytes contain the counter values for the timer on the STM8S. The counter values determine the number of high / low sequences and their duration.

One important consideration is the length of the command name. If the name is too short then the name is not meaningful to the user, too long and the number of commands is reduced and the STM8S may become unusable.

Editing Pulses in the Configuration Application

The lower part of the configuration application contains a section for the creation, modification and deletion of commands in the remote control:

OpenIR Main Form

OpenIR Main Form

Each remote control command is a series of pulses (IR signal is on) and spaces (IR signal is off). Each on /off sequence can be represented by a number of microsecond periods, the sequence starts with an on pulse and each subsequent number represents a change from on to off and so on.

Editing the wave form has been facilitated with a simple form containing a list box to hold the pulse durations and a user control presenting a graphical representation of the pulse waveform:

Add Pulse Sequence

Add Pulse Sequence

The above waveform is the sequence required to activate the shutter on a Nikon camera.

Clicking on the Add button adds the name to the main form.

Main Form With Command

Main Form With Command

Clicking on the Show EEPROM displays the raw bytes in the EEPROM:

EEPROM Memory With Pulse Data

EEPROM Memory With Pulse Data

Conclusion

The changes to the application now allow for the storing of sequences in the EEPROM. Each command is given a name which is meaningful to the user.

The current implementation is not perfect and a number of changes are in the pipeline:

  1. The pulse sequences are currently stored as microsecond values. Exact counter values for Timer 2 will make the STM8S application smaller.
  2. Pulses are stored in the EEPROM s bytes but the configuration application uses ArrayLists and List objects for the command edit forms. It may be possible to provide a more elegant method for moving this data around the application.

EEPROM Memory Dump

February 15th, 2015 • Software Development, STM83 Comments »

One debugging feature I have been keen to add is the ability to see the EEPROM memory. This will aid the debugging of the code on the STM8S as it will be easier to see the data being consumed by the application running on the remote control. The EEPROM memory dump feature simply displays a grid of memory locations along with the contents.

Main Form

The main form has been modified to take values from the user and then to translate the contents of the user controls into data for the EEPROM.

For example, the carrier frequency needs to be translated into counter values for the PWM function in Timer 1. This could be sent over to the remote control as a frequency but then it would not be possible to verify if the remote control could generate the desired frequency until the remote control tried to use the values. If the Windows application is going to perform checks on the user input then it is logical that it should send over just the counter values and not the frequency. This offloads the code necessary to perform the translation from the remote control to the user application. Doing this makes the application on the STM8S smaller. Remember, we have an 8K code limit on the remote control when using the IAR compiler.

The first step is to make the controls on the user interface respond to the values being entered and also add a mechanism to show the form which will display the EEPROM memory:

OpenIR Main Form

OpenIR Main Form

EEPROM Class

The application will also need a class in order to hold the contents of the EEPROM. This class acts as an intermediary between the raw bytes in the EEPROM and the data displayed in the user controls. The properties in the class translate the bytes into data types which can be used by a C# application and visa versa.

For instance, consider the remote control name. The C# application would like to see a C# string object. In the remote control EEPROM this is a series of up to 16 bytes each holding one character. A property presents a convenient way of performing this translation. The EEPROM class will require a series of bytes to hold memroy contents:

private byte[] _memory;

The Name property can then translate the C# interface requirements into those required by the STM8S application:

public string Name
{
    get
    {
        string result;

        if (_memory != null)
        {
            int index = NAME_OFFSET;
            result = "";

            while ((index < NAME_LENGTH) && (_memory[index] != 0))
            {
                result += (char) _memory[index++];
            }
        }
        else
        {
            result = null;
        }
        return (result);
    }
    set
    {
        if (_memory == null)
        {
            _memory = new byte[EEPROM_LENGTH];
        }
        if (value.Length > NAME_LENGTH)
        {
            throw new ArgumentOutOfRangeException("Name", string.Format("Name must be less than {0} characters in length.", NAME_LENGTH));
        }
        for (int index = NAME_OFFSET; index < (NAME_OFFSET + NAME_LENGTH); index++)
        {
            if (index < value.Length)
            {
                _memory[index] = (byte) value[index];
            }
            else
            {
                _memory[index] = 0;
            }
        }
    }
}

Similar properties can be added for the carrier frequency etc.

EEPROM Form

The form displaying the EEPROM memory makes use of a Grid control in the Syncfusion Essential Studio Enterprise Edition Community edition. This suite of tools contains 650+ controls for a variety of platforms and has recently been made available at no charge to individual developers and small organisations with a low turnover (< $1m).

The form showing the EEPROM memory is simple and contains the memory contents and a button to close the form

EEPROM Memory Dump Form

EEPROM Memory Dump Form

The image above shows the memory when the following properties have been set:

  1. Name of the remote control (row 0x000, 16 bytes)
  2. Counter value for the PWM function on Timer1 (row 0x0010, first two bytes)
  3. Power LED status (row 0x0010, offset 5)

Conclusion

Development of the Windows interface is proceeding steadily. As much work as possible is being offloaded to the Windows application in order to streamline the code which needs to be written for the STM8S.

OpenIR Bidirectional Communication

February 1st, 2015 • Software Development, STM8Comments Off

Progress on the OpenIR project has been a little slow recently, Christmas has come and gone and now a heavy workload is slowing things down further. Having said that, today has seen the project pass another milestone with a Windows configuration application communicating with the an STM8S Discovery board over a TTL serial port.

This post will give an overview of the current progress.

Windows Configuration Application

One of the goals of the OpenIR project is to create a universal remote control. To this end the project will require a configuration application. Having a number of years experience in Windows programming it made sense for the Windows platform to host the first generation of the configuration application for the remote control.

In the previous post a number of command functions were identified as being essential to this project. The main concern for the initial development is the size of the application which IAR can support on the STM8S. This is limited to 8KB and the EEPROM transfer function is likely to consume the most memory and code space. This function has been targeted first as it is likely to identify any issues early on in the software development phase of the project.

The Windows configuration application is a classic WinForms application with three distinct areas:

  1. Communication settings (serial port)
  2. General configuration
  3. Commands (IR sequences)

This currently looks as follows:

Windows Configuration Application

Windows Configuration Application

The upper panel allows the user to select the COM port and the communication settings (baud rate, parity etc.). The two buttons allow the user to request the EEPROM data and write send the updated the EEPROM configuration back to the IR remote control module.

The middle section contains the controls which will show and allow the editing of the static configuration such as the name, carrier frequency etc.

The lower panel contains the command list the remote control can send. More on this in a future port.

The current application allows the communication settings to be changed and implements the Read EEPROM request.

Data Packet Format

The initial design of the data packets allows for a request or response to be transferred with an optional data packet. The basic format is as follows:

Offset Length Description
0 1 Data packet header (0xaa).
1 2 Length of the data packet (unsigned short), high byte first.
3 1 Command to be executed.
4 n Data required for this command.
4 + n 1 Checksum for the entire packet.

The packet header is an arbitrary value and 0xaa has been chosen as it is an alternating sequence of bits.

The initial design packet size was expected to be less than 256 bytes. As the design progressed it became apparent that is was desirable for the packets to be greater larger than 256 bytes.

There are a limited number of commands which have been identified for this project. At the current time this is set to be 7 and a single byte is sufficient.

The data packet is optional and in the case of the EEPROM read/write functions this will be the contents of the EEPROM either being read or written.

The checksum byte is a simple exclusive OR of all of the bytes in the packet from the initial packet header through the the end of the data packet. The starting value for the checksum is 0xaa.

The configuration application will first scan the PC for COM ports. Any available ports will be added to the drop down list of COM port names. Selecting a COM port will populate the fields with the default COM port configuration.

Clicking on the Read EEPROM button send a request to the STM8S. The STM8S will respond with the contents of the EEPROM. This can be seen in the following traces from the logic analyser.

The first trace shows the request packet (top trace) and the response from the IR remote control (lower trace):

EEPROM Request and Exchange

EEPROM Request and Exchange

Zooming on on the request trace we can see that the command 1 (request for EEPROM data) is sent to the IR remote control:

EEPROM Request

EEPROM Request

Moving along the timeline the lower trace shows the response from the module expanded:

EEPROM Data

EEPROM Data

Conclusion

The initial communications with the IR remote control and a PC has been successful. At this point in time the source code requires some clean up work. The next step is to enhance the EEPROM configuration allowing the EEPROM to be rewritten upon command from the Windows configuration application.

OpenIR Design Update

January 4th, 2015 • Electronics, Software Development, STM8Comments Off

Revision A of the board is now working and can send a single IR sequence out to a device in the real world when the on board switch is pressed. If OpenIR is to be truly universal the system needs to be able to send a multitude of commands not just a single command. In order to do this we need to be able to store IR command sequences and also allow the user to select which IR sequence is transmitted.

The STM8S has been set up to connect the TTL serial port to the FTDI and RedBear BLE board ports. Doing this allows communication with the outside world (PC, iPhone etc.). The proposed solution uses the serial TTL port to send commands to the STM8S and for the STM8S to store details of the IR signals (carrier frequency, active period etc.) in the on chip EEPROM.

The chip along with the chosen have a limit built into them, the fact that the free version of the IAR tools have an 8 KByte limit. This limits what can be achieved on the STM8S microcontroller.

Serial Commands

The STM8S will listen on the serial TTL port for commands from the outside world. The following list of commands are proposed as a starting point:

Command ID Description
1 Get Remote control ID. This returns a text string which identified the remote control.
2 Set the remote control ID.
3 Get the carrier frequency. This gets the two bytes which are used by Timer 1 to determine the frequency of the PWM signal.
4 Set the carrier frequency. This set the two bytes which are used by Timer 1 to determine the frequency of the PWM signal.
5 Get the contents of the EEPROM pulse data store.
6 Set the contents of the EEPROM pulse data store on the STM8S.
7 Transmit pulses for sequence number x where x is the item in the payload.
8 Transmit pulses. This transmits and arbitrary sequences of pulses which are contained in the remainder of the payload.
9 Time Lapse mode. Send the pulses for sequence x after y seconds.
10 Reset the remote control.
11 Enable or disable the on board power LED.

A close look at the above shows that commands 1, 3 and 4 are related as are commands 2, 4 and 6. They are either getting or setting blocks of memory in the STM8S EEPROM. Given the reduced memory available and the limits of the tools it may be optimal reduce this to reading and writing the contents of the EEPROM. The configuration data would be processed on a device with more memory (PC, iPhone etc.) and the EEPROM image built and transmitted to the STM8S. The STM8S then simply needs to update the EEPROM. The final command set becomes:

Command ID Description
1 Get the contents of the EEPROM pulse data store.
2 Set the contents of the EEPROM pulse data store on the STM8S.
3 Transmit pulses for sequence number x where x is the item in the payload.
4 Transmit pulses. This transmits and arbitrary sequences of pulses which are contained in the remainder of the payload.
5 Time Lapse mode. Send the pulses for sequence x after y seconds.
6 Reset the remote control.
7 Enable or disable the on board power LED.

Layout of the EEPROM

The STM8S on the EEPROM stores the configuration of the remote control. The data stored is a mixture of basic configuration along details of the pulses for each command the remote control can transmit.

Offset Length Description
0x00 16 Text ID of the remote control
0x10 2 Two bytes which are used by Timer 1 to determine the frequency of the carrier signal. The carrier signal is assumed to be 50% duty cycle.
0x12 1 Number of command sequences stored in the EEPROM.
0x13 1 Number of seconds to use for the time lapse sequence.
0x14 12 Unused.
0x20 64 Length of the pulse sequences (0x20 = length of sequence 0, 0x21 = length of sequence 1 etc.).
0x60 512 Pulse data. It is assumed that the pulse sequences will start with an on period followed by and off period until the number of sequences have been consumed.

Conclusion

The basic layout of the EEPROM has been determined along with a proposed command sequence. The next step is to implement the STM8S code and some sample Windows code to configure the remote control.

Connecting Informatica to Twitter

December 29th, 2014 • InformaticaComments Off

In the past few months I have started to use Informatica Power Centre at work. During the course the instructor mentioned that Informatica had released an Express version of the full development tool. This is fully feature but limited in the number of records it can process. The limit is 250,000 records per day so this makes it useful for personal, non-commercial use. This has lead me to install the system on my home computer and to start to investigate some of the features I will not have the opportunity to use in my day job.

Informatica Power Centre Express can be connected to several data sources including social media site such as Twitter and Facebook. I am highly unlikely to have the opportunity to use Informatica in this area and so a weekend project came into being, the problem, how to connect Power Centre Express to Twitter, retrieve a list of followers and store this in a local database.

Problem Definition

Informatica Power Centre Express can performed a number of operations with Twitter:

  • Search
  • Lookup a profile
  • Get a list of Followers for the account
  • Get a list of Friends for the account

The two operations of interest are the Get list of Followers and Profile Lookup functions.

The general algorithm is as follows:

  1. Get a list of Followers for the Twitter user
  2. Compare the list to those in the local database.
  3. If the Twitter user ID is found then update the record to indicate that the user is still a follower at the time the list was retrieved.
  4. If the Twitter user is not in the list of Followers in the local database then retrieve information about the Twitter user and add to the local database.

One of the overriding aims of this is to only ever retrieve as much information as necessary and never any more. It is therefore essential that the data is filtered as soon as possible to prevent retrieving information which we already hold locally.

Connecting Power Centre Express to Twitter

Rather then duplicate existing knowledge, I draw your attention to the article Informatica PowerCenter Express – Connecting to Twitter. This is a fairly comprehensive article but it stops short of what will be achieved in this post, namely dynamic lookup of user information.

In order to implement the above article you will need to have the following completed:

  1. Download and install Power Centre Express
  2. Create a Twitter account and also register for a Twitter developer ID

At the end of the article you should have a Power Centre mapping which can retrieve static information from Twitter.

Retrieving Dynamic Data

If you have followed the above article you will be able to retrieve information about a user, a number of posts which match search criteria and lookup a profile. This is all achieved by changing the Query Parameter in the Twitter_Data_Object_Operation and running a mapping through the Developer user interface.

Hidden away in the help system is the key fact which we need to allow the retrieval of multiple user profiles where the user IDs are taken from a dynamic data source, namely a text file.

Twitter Project

The Informatica Twitter Project created following the guidelines in the above article looks like this:

Informatica Twitter Project Treeview

Informatica Twitter Project

This is slightly different from the project in the article as the project is intended to process the data retrieved.

conInformatica

This object is a connection from Informatica to a SQL Server local database.

rdoTwitterUser

This object is a relational database object in the local SQL Server database. It contains the mapping to a database table which can be used to hold information about the Twitter user. The SQL code to create a small table to hold the Twitter user details is as follows:

CREATE TABLE [dbo].[TwitterUser]
(
	[ID] [int] IDENTITY(1,1) NOT NULL,
	[Description] [varchar](512) NULL,
	[TwitterID] [varchar](160) NULL,
	[IdStr] [varchar](160) NULL,
	[ScreenName] [varchar](160) NULL,
	[LastUpdated] [datetime] NULL DEFAULT (getdate())
) ON [PRIMARY]

This table holds a local ID, the Twitter ID (TwitterID and IdStr), Twitter account name (ScreenName), a description (Description) for the account and the date/time the record in the local database was last updated (LastUpdated).

When executing a Twitter Profile Lookup, the Twitter API returns much more information than the above. The following creates a table which would hold all of the information returned from the API:

CREATE TABLE [dbo].[TwitterUser]
(
	[ID] [int] IDENTITY(1,1) NOT NULL,
	[ContributorsEnabled] [bit] NULL,
	[CreatedAt] [datetime] NULL,
	[DefaultProfile] [bit] NULL,
	[DefaultProfileImage] [bit] NULL,
	[Description] [varchar](512) NULL,
	[FavouritesCount] [bigint] NULL,
	[FollowRequestSent] [bit] NULL,
	[FollowersCount] [bigint] NULL,
	[Following] [bit] NULL,
	[FriendsCount] [bigint] NULL,
	[GeoEnabled] [bit] NULL,
	[TwitterID] [varchar](160) NULL,
	[IdStr] [varchar](160) NULL,
	[IsTranslator] [bit] NULL,
	[Lang] [varchar](40) NULL,
	[ListedCount] [bigint] NULL,
	[Location] [varchar](360) NULL,
	[Name] [varchar](220) NULL,
	[Notifications] [bit] NULL,
	[ProfileBackgroundColour] [varchar](120) NULL,
	[ProfileBackgroundImageURL] [varchar](940) NULL,
	[ProfileBackgoundImageURLHttps] [varchar](940) NULL,
	[ProfileBackgroundImageTile] [bit] NULL,
	[ProfileImageURL] [varchar](1600) NULL,
	[ProfileImageURLHttps] [varchar](1640) NULL,
	[ProfileLinkColour] [varchar](120) NULL,
	[ProfileSidebarBorderColour] [varchar](120) NULL,
	[ProfileSidebarFillColour] [varchar](120) NULL,
	[ProfileTextColour] [varchar](120) NULL,
	[ProfileUseBackgroundImage] [bit] NULL,
	[Protected] [bit] NULL,
	[ScreenName] [varchar](160) NULL,
	[ShowAllInlineMedia] [varchar](100) NULL,
	[StatusContributors] [varchar](512) NULL,
	[StatusCoordinates] [varchar](512) NULL,
	[StatusCreatedAt] [datetime] NULL,
	[StatusFavourited] [bit] NULL,
	[StatusGeo] [varchar](512) NULL,
	[StatusID] [varchar](360) NULL,
	[StatusInReplyToScreenName] [varchar](512) NULL,
	[StatusInReplyToStatusID] [varchar](512) NULL,
	[StatusInReplyToUserID] [varchar](512) NULL,
	[StatusPlace] [varchar](512) NULL,
	[StatusPossiblySensitive] [varchar](100) NULL,
	[StatusRetweetCount] [bigint] NULL,
	[StatusRetweeted] [bit] NULL,
	[StatusSource] [varchar](1440) NULL,
	[StatusText] [varchar](1900) NULL,
	[StatusTruncated] [bit] NULL,
	[StatusesCount] [bigint] NULL,
	[TimeZone] [varchar](120) NULL,
	[URL] [varchar](480) NULL,
	[UTCOffset] [bigint] NULL,
	[Verified] [bit] NULL,
	[LastUpdated] [datetime] NULL DEFAULT (getdate())
) ON [PRIMARY]

conNevynTwitterAccount

This connection object connects the Informatica project to a Twitter account.

tdoNevynsTwitter

tdoNevynsTwitter is a Twitter Data Object connected to a Twitter account.

adoGetFollowers

This data object gets a lit of Followers for the Twitter account. It retrieves a list Twitter user IDs and nothing more.

adoGetUserDetails

adoGetUserDetails retrieves the details for one or more users. In the article above it retrieved information about a single user by setting the query parameter to something like user_id=123456 where 123456 represents the Twitter user ID to lookup.

The Query Parameter can be modified to read the IDs from a text file by setting the value to something like user_id=file:///c:\tmp\UserIDs.txt.

Nevyn's Twitter Data Object

Nevyn’s Twitter Data Object

Informatica will now use the file as the data source for the query.

ffUserIDs

ffUserIDs is a flat file object which will contain the Twitter IDs of the Followers to look up. This will be the same file specified in the Query Parameter of the adoGetUserDetails object.

Mappings

Two mappings are required to achieve the project goals, the first mapGetFollowers retrieves the full list of Followers for the Twitter account. The second mapping, mapUpdateTwitterUsersTable, looks up the new Followers and adds them to the local database.

mapGetFollowers

mapGetFollowers retrieves the full list of Followers for the Twitter account.

Get Followers Mapping

Get Followers Mapping

The User IDs are split into two groups by the mapping, the first group contains the IDs of the new Followers. These IDs are written to the flat file object ffUserIDs. The second group are used to update the local database to say that the IDs stored are still Followers on the date the mapping was run.

adoGetFollowers retrieves the full list of Followers for the Twitter account.

The list of Follower IDs are then used in lkpUser to see if the FollowerIDs retrieved exist in the local database. The IdStr field in the lookup will contain the ID of the Twitter user for Followers whose IDs are in the local database and will be null for new Followers.

expAddLastUpdatedField simply adds the current date/time to the record set.

rtrSplitExistingFromNewFollowers splits the records into two groups, new Followers and existing Followers. The IDs of new Followers are written to the flat file ffUserIDs. The IDs for existing followers are passed to the update strategy updExistingFollowers, this will update the local database by matching the Follower ID and updating the LastUpdated column in the database.

mapUpdateTwitterUsersTable

mapUpdateTwitterUsersTable reads the Follower IDs from the flat file ffUserIDs and passes them to Twitter in order to get the profile for the users. Twitter will send back the profile for each of the user IDs in the file.

Update Twitter Users Table Mapping

Update Twitter Users Table Mapping

The Required data is then extracted and converted from Twitter data (strings etc.) into SQL data types by the expression expConverTwitterDataToSQLDataTypes.

wfUpdateFollowersInformation

This workflow allows the two mappings to be executed:

Update Followers Information Workflow

Update Followers Information Workflow

The command tasks in the workflow simply audit the execution of the mappings recording start times, number of records etc. into a log file.

Data Types and Conversions

The current project simply looks at the number of Followers and the Twitter account names. Along the way the full data set returned by Twitter was examined and conversions for the various fields put together. If you wish to extend this project then you may find the following conversions useful.

Data Type Conversion/Expression
Boolean IIF(LOWER(boolean_field) = ‘true’, 1, 0)
BigInt IIF(IS_NUMBER(big_int_field), TO_BIGINT(big_int_field), 0)
Date / Time IIF(ISNULL(date_field), NULL, to_date(substr(date_field, length(date_field) – 3) || ‘-‘ || substr(date_field, 5, 3) || ‘-‘ || substr(date_field, 9, 2) || ‘ ‘ || substr(date_field, 12, 8)))

Conclusion

The free Express edition of Informatica Powercentre allows the home user to experiment with the full power of Informatica. The social media connectors allow data to be retrieved and analysed locally as in this case.

If you’ve found this useful and wish to use the above you should remember to read the terms and conditions of the Twitter API and remember to respect the privacy of Twitter users.

Teensy 3.1 and Visual Micro

October 14th, 2014 • Electronics, Software DevelopmentComments Off

September saw the end of an agonising few months (nearly a year in truth) working with a 32 x 32 LED matrix and smart LEDs (WS2811, Neopixels etc.). I’ve been trying to reliably control these LEDs with the STM32 Discovery board. It’s not that I cracked the problem with the STM32 but that I came across a cheap ready made solution to the problem, namely the Teensy 3.1. It has proved to be a painful reminder of something a software engineer should know, use libraries if you can. In the case of hardware prototyping the use of libraries also extends in part to the choice of hardware.

One of the main reasons I have resisted going down the route of using boards like the Teensy 3.1 is the Arduino IDE. I think that I have been spoiled by the richness of the Visual Studio IDE. Even IDES should as the IAR IDE for the STM8S leave something to be desired when you compare it to Visual Studio. This is the story of how I got the LEDs working and solved the IDE issue by using Visual Micro.

Problem Hardware

There are two pieces of hardware causing me problems at the moment:

  1. 32 x 32 LED Matrix
  2. 5mm Digital Addressable LEDs

These were purchased in the UK from Cool Components.

32 x 32 LED Matrix/Panel

The 32 x 32 LED matrix is often a component in larger displays.

LED Matrix

LED Matrix

The displays are easily chained together although the power requirements quickly increase. Each panel can consume up to 2A depending upon the number of LEDs illuminated as any time.

These boards are designed to be controlled by FPGAs allowing for high frame rates. Sparkfun and Adafruit have tutorials on running these boards and they have found that you can run a single panel at 16MHz (i.e. you can run it on an Arduino but only just). Running two or more panels requires more power, both RAM and processor speed.

Digitally Addressable LEDs

The LEDs in question use the WS2811 constant current IC to drive RGB LEDs. These controllers use their own one-wire protocol (not to be confused with the Dallas one-wire protocol) to determine the PWM characteristics of the red, green and blue components of the light output. Over the years these controllers have shown up in LED strings, LED strips and more recently as the Adafruit Neopixel and now as individual through hole LEDs.

The main problem I have found with these LEDs, or more specifically the controllers, is the one-wire protocol. This requires very precise timing and this is often difficult to obtain on the hobbyist microcontrollers without a lot of very precise control.

Solving the problem

I decided to solve this problem using the Teensy 3.1. The thing which makes the Teensy 3.1 very attractive for this type of project is the add-on board, the SmartMatrix Shield. This board allows you to connect a Teensy directly on to the IDC headers of the LED Matrix.

In addition to the hardware, the Teensy 3.1 library contains ports of the equivalent Arduino libraries for the 32 x 32 LED matrix as well as LEDs controlled by the WS2811.

Teensy 3.1

The Teensy has been through several iterations, the Teensy 3.1 being the latest at the time of writing. The board is small, only 35mm x 18mm, and I was not really prepared for how small it actually is!

TeensyAndRuler

Small does not mean low powered though, the board packs a Cortex-M4 processor, 256K flash and 64K RAM. The 72MHz processor can also be overclocked to over 90MHz. Other features include:

  • 34 Digital I/O (3.3V but 5V tolerant)
  • 21 analog inputs
  • 12 PWM
  • SPI, I2C, CAN bus
  • ARM DSP extension

The foot print of the board allows it to be easily added to a breadboard.

The Teensy 3.1 board is also supplied with it’s own libraries in the form of Teensyduino. This is an add-on for the Arduino IDE and is compatible with many of the Arduino libraries. The Arduino compatible libraries provide the ability to work with the LED matrix and the addressable LEDs mentioned above as well as a wide variety of other devices.

Software

As I have stated earlier, I am not really a fan of the Arduino IDE. Do not misinterpret me, it represents a great piece of work, however the amount of time and money Microsoft have poured into Visual Studio over the years means that Visual Studio really does take some beating. This is where Visual Micro comes in.

Visual Micro is a Visual Studio 2008-2013 add-in which allows the development of Arduino sketches within the Visual Studio framework. This gives you all the power of Visual Studio (code refactoring etc.) while still allowing the development of Arduino sketches. The add-in is free until the end of 2014 with the option to purchase the debugger support for a nominal fee.

The add-in supports a good number of boards out of the box including the Teensy 3.1.

Setup is simple and the documentation is really good. There are a few additional steps for the Teensy 3.1 but this is cover in the Tips for Teensy page.

Once configured it was a simple task to create a new Teensy 3.1 project using the Neopixel sketch template. A little code reformatting gives the following code:

#include <Adafruit_NeoPixel.h>

//
//	Parameter 1 = number of pixels in strip
//	Parameter 2 = pin number (most are valid)
//	Parameter 3 = pixel type flags, add together as needed:
//		NEO_RGB     Pixels are wired for RGB bitstream
//		NEO_GRB     Pixels are wired for GRB bitstream
//		NEO_KHZ400  400 KHz bitstream (e.g. FLORA pixels)
//		NEO_KHZ800  800 KHz bitstream (e.g. High Density LED strip)
//
Adafruit_NeoPixel strip = Adafruit_NeoPixel(60, 6, NEO_RGB + NEO_KHZ800);

//
//	Input a value 0 to 255 to get a color value.
//	The colours are a transition r - g - b - back to r.
//
uint32_t Wheel(byte WheelPos)
{
	if (WheelPos < 85)
	{
		return strip.Color(WheelPos * 3, 255 - WheelPos * 3, 0);
	}
	else if (WheelPos < 170)
	{
		WheelPos -= 85;
		return strip.Color(255 - WheelPos * 3, 0, WheelPos * 3);
	}
	else
	{
		WheelPos -= 170;
		return strip.Color(0, WheelPos * 3, 255 - WheelPos * 3);
	}
}

//
//	Fill the dots one after the other with a colour
//
void colorWipe(uint32_t c, uint8_t wait)
{
	for (uint16_t i = 0; i < strip.numPixels(); i++)
	{
		strip.setPixelColor(i, c);
		strip.show();
		delay(wait);
	}
}

//
//  Set all LEDs to the same colour.
//
void rainbow(uint8_t wait)
{
	uint16_t i, j;

	for (j = 0; j < 256; j++)
	{
		for (i = 0; i < strip.numPixels(); i++)
		{
			strip.setPixelColor(i, Wheel((i + j) &amp; 255));
		}
		strip.show();
		delay(wait);
	}
}

//
//	Slightly different, this makes the rainbow equally distributed throughout
//
void rainbowCycle(uint8_t wait)
{
	uint16_t i, j;

	for (j = 0; j < 256 * 5; j++)
	{ // 5 cycles of all colors on wheel
		for (i = 0; i < strip.numPixels(); i++)
		{
			strip.setPixelColor(i, Wheel(((i * 256 / strip.numPixels()) + j) &amp; 255));
		}
		strip.show();
		delay(wait);
	}
}

//
//	Executed once at startup to set up the board.
//
void setup()
{
	strip.begin();
	strip.show(); // Initialize all pixels to 'off'
}

//
//	The actual main program loop.
//
void loop()
{
	rainbow(20);
	rainbowCycle(20);
}

This code creates an Adafruit_NeoPixel object with the pixels connected to pin 6 of the Teensy 3.1.

Connecting some LEDs top the Teensy and uploading the above code results in the following:

Modifying the code to add a counter will allow testing of the debugging features. The loop code is changed to have a counter variable to test the conditional breakpoint feature:

int count = 0;

//
//	The actual main program loop.
//
void loop()
{
	count++;
	digitalWrite(LED_PIN, HIGH);
	delay(500);
	digitalWrite(LED_PIN, LOW);
	delay(500);
	// Some example procedures showing how to display to the pixels:
	rainbow(20);
	rainbowCycle(20);
}

Adding a breakpoint after the counter increment and making the breakpoint conditional results in the following display:

Sample code

Sample Code With Breakpoint

Nothing too strange here, pretty much standard for Visual Studio. The breakpoint window shows the breakpoint and the condition:

Debugger Breakpoint Window

Debugger Breakpoint Window

Running the code brings up the debugger expression window:

Debugger Expression Window

Debugger Expression Window

The one thing which experienced users of Visual Studio will find unfamiliar is the way in which single stepping works. Unlike say Win32 development where single stepping takes you to the next line of code and then waits, with Visual Micro, single stepping runs the code to the next breakpoint.

Conclusion

Visual Studio is without doubt one of the best software development environments available. The addition of Visual Micro brings the power of Visual Studio to the Arduino world. Whilst the debugging features (single stepping) may a little unfamiliar to Visual Studio users they are not so strange as to be unusable.

Both the Teensy 3.1 and Visual Micro are a welcome addition to the development tools and it is highly likely that they will appear in future posts.

Using the Electric Imp

September 1st, 2014 • Electronics, Internet of Things, Software DevelopmentComments Off

A few weeks ago I acquired and Electric Imp as I was interested in how this could be used to prototype and connect a device to the Internet. Such a device would become part of the Internet of Things.

In order to investigate the possibility of using this device I decided to monitor the temperate of my office and log the data on the Internet.

Electric Imp

Electric Imp offers a starting point for hardware and software engineers wishing to develop for the Internet of Things. The simplest format for prototyping is probably the Imp001 and a breakout board. The Imp001 is an Electric Imp in SD format. The card is fully FCC certified and so offers a simple way of using a wireless network to connect a project to the Internet. The SD card contains a Cortex-M3 processor, 802.11b/g/n transceiver and antenna and once connected to a WiFi network with internet connectivity can be programmed using the cloud based development environment.

The Electric Imp is connected to a WiFi network using an application called BlinkUp. This application is a free download for iPhone and Android. The phone application takes details about the local WiFi network (name, password etc.) and sends this to the Imp001 by blinking the screen (hence the name BlinkUp) whilst the phone is held against the LED on the SD card.

Software for the Electric Imp is developed using the online IDE provided by Electric Imp. The development environment offers the ability to develop code which runs on the Electric Imp (Device code) itself as well as a component which runs in the cloud (Agent code). A user account is setup by following the link to the Log in page from the Electric Imp web site.

Sparkfun Data Logging Service

Sparkfun have recently started to provide a cloud based data logging service which is free to use for a limited amount of data. The system uses a circular 50 MByte data store for each data stream. Sign up is simple, just follow the Create link from the main page. The data streams can be both public and private. If the amount of data storage required is greater than 50MB or the application requires a greater level of privacy then the source is freely available for download by following the DFeploy link from the main page.

Once created, the data stream is accessed using public and private keys, the private keys allow data to be written to the data stream. The public key allows the public data stream to be viewed or data to be retrieved for use in say charting.

Temperature Logging

The principles involved in linking the local hardware to the Sparkfun cloud server will be illustrated by logging the local temperature using a temperature sensor and then sending the data to Sparkfun’s servers. The data will be retrieved and displayed on a web page.

Hardware

The bill of materials (BOM) for this project is as follows:

  1. Electric Imp (Imp001)
  2. Electric Imp Breakout board
  3. LM35 Temperature Sensor
  4. LED and current limiting resistor
  5. Connectors
  6. Breadboard and miscellaneous wire
  7. USB cable and power supply (your computer can act as a power supply if necessary)

Solder the connectors to the breakout board and insert the connectors into the breadboard.

Next, connect the current limiting resistor to Pin 9 on the breakout board and the LED. Connect the other leg of the LED through to ground.

Finally connect the LM35 temperature sensor to Vcc, Ground and the sensor output to Pin 2 of the breakout board.

Follow the instructions on the Electric Imp web site for downloading the BlinkUp software and configuring the Electric Imp. Configure the Imp and connect to the local WiFi network.

Electric Imp Software

The Electric Imp software is split into two components:

  1. Device code running on the Electric Imp hardware
  2. Agent code which runs on the Electric Imp cloud servers

The first step is to register with the Electric Imp web site for a developer account. Once completed you will be presented with the developer IDE.

To record the temperature the Electric Imp will record the temperature every 5 seconds. These readings will be summed and averaged over a one minute period. The average will sent to the Electric Imp servers, the average cleared and the whole process will restart. This will provide a continuous stream of temperature readings while the Imp is powered.

The Electric Imp servers will run Agent code which will listen for data/commands from the Electric Imp device code. The Agent on the server will then post the data to the Sparkfun server.

The code for the Device and the Agent is written in a C like language called Squirrel.

Device Code

The analog port on the Electric Imp returns a value in the range 0 to 65,535. The maximum value represents a voltage of 3.3V. The temperature sensor selected outputs 10mv per degree C. The maximum range of the values for this sensor is 0V to 1.55V given the operating range for the LM35.

The LED has been added to demonstrate when the board is taking a temperature reading. The LED will flash each time a reading is taken.

Firstly, some space in needed for the supporting variables:

//
//  Create a global variable to allow control of the LED.
//
led <- hardware.pin9;
//
//  Create a global variable for the temperature sensor.
//
temperatureSensor <- hardware.pin2;
// 
//  Configure led to be a digital output.
//
led.configure(DIGITAL_OUT);
//
//  Configure the temperature sensor to be an analog input.
//
temperatureSensor.configure(ANALOG_IN);
//
//  This name will be sent to the stream each update:
//
local impName = "Imp%20" + imp.getmacaddress();
//
//  Variables related to averaging the temperature.
//
readingCount <- 0;
readingSum <- 0.0;
// 
//  Create a global variables for the sensor readings.
//
temperature <- 0.0;
sensorValue <- 0;

Next, the main application loop (function), the first thing this should do is to turn on the LED to show that the application is active:

//
//  Main program loop.
//
function main()
{
    led.write(1);

Next, take the temperature sensor reading and convert to centigrade and add to the ongoing sum:

    sensorValue = temperatureSensor.read();
    temperature = ((sensorValue * 3.3) / 65535) * 100;
    readingCount++;
    readingSum += temperature;

Next, check if the number of readings has reached 12 (60 seconds). If we have 12 readings then take the average and send this to the Electric Imp Agent:

    if (readingCount == 12)
    {
        local average = readingSum / readingCount;
        server.log("Average temperature = " + average);
        local data = "";
        data = "Temperature=" + average;
        agent.send("postData", data);
        readingCount = 0;
        readingSum = 0.0;
    }

The server.log statement sends the logging information to the servers. This is not used anywhere, simply logged. The data is sent to the Agent in the agent.send(“postData”, data) statement.

Next, pause and then turn the LED off:

    imp.sleep(0.5);
    led.write(0);
</p>
<p>
The whole process should be repeated 4.5 seconds later (remember there is a 0.5 seconds pause above) to take readings every 5 seconds.
</p>
<p>
[code="c"]
    //
    //  Schedule imp to wakeup and repeat.
    //
    imp.wakeup(4.5, main);
}

Finally the main loop should be executed:

//
//  Start the main program loop.
//
main();

Agent Code

The Agent code is responsible for listening for data from the device. The code used is actually provided by Sparkfun and is produced here more or less unaltered:

/*****************************************************************
Phant Imp (Agent)
Post data to SparkFun's data stream server system (phant) using
an Electric Imp
Jim Lindblom @ SparkFun Electronics
Original Creation Date: July 7, 2014

Description

Before uploading this sketch, there are a number of vars that need adjusting:
1. Phant Stuff: Fill in your data stream's public, private, and 
data keys before uploading!

This code is beerware; if you see me (or any other SparkFun 
employee) at the local, and you've found our code helpful, please 
buy us a round!

Distributed as-is; no warranty is given.
*****************************************************************/

//
//  Phant Stuff configuration information.
//
local publicKey = "Your-Public-Key-Goes-Here";      // Your Phant public key
local privateKey = "Your-Private-Key-Goes-Here";    // Your Phant private key
local phantServer = "data.sparkfun.com";            // Your Phant server, base URL, no HTTP

//
//  When the agent receives a "postData" string from the device, use the
//  dataString string to construct a HTTP POST, and send it to the server.
//
device.on("postData", function(dataString)
    {
        server.log("Sending " + dataString); // Print a debug message
        //
        //  Construct the base URL: https://data.sparkfun.com/input/PUBLIC_KEY:
        //
        local phantURL = "https://" +  phantServer + "/input/" + publicKey;
        //
        //  Construct the headers: e.g. "Phant-Private-Key: PRIVATE_KEY"
        //
        local phantHeaders = {"Phant-Private-Key": privateKey, "connection": "close"};
        //
        //  Send the POST to phantURL, with phantHeaders, and dataString data.
        //
        local request = http.post(phantURL, phantHeaders, dataString);
        //
        //  Get the response from the server, and send it out the debug window:
        //
        local response = request.sendsync();
        server.log("Phant response: " + response.body);
    }
);

The code is clearly commented and self explanatory.

Results

Building the above in the IDE should deploy the device code to the Electric Imp should result in the temperature being collected by the device, sent to the Electric Imp server and then from there on to Sparkfun’s servers. The data can be viewed in it’s raw form by browsing to your data stream using a URL such as: https://data.sparkfun.com/streams/Your-Public-Key-Here. This URL will have been supplied on the account creation page for the Sparkfun data service.

The data can also be retrieved using Javascript:

<!DOCTYPE html>
<html>
  <head>
    <!-- EXTERNAL LIBS-->
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
    <script src="https://www.google.com/jsapi"></script>

    <!-- EXAMPLE SCRIPT -->
    <script>

      // onload callback
      function drawChart() {

        var public_key = 'Your-Public-Key-Here';

        // JSONP request
        var jsonData = $.ajax({
          url: 'https://data.sparkfun.com/output/' + public_key + '.json',
          data: {page: 1},
          dataType: 'jsonp',
        }).done(function (results) {

          var data = new google.visualization.DataTable();

          data.addColumn('datetime', 'Time');
          data.addColumn('number', 'Temperature');

          $.each(results, function (i, row) {
            data.addRow([
              (new Date(row.timestamp)),
              parseFloat(row.Temperature),
            ]);
          });

          var chart = new google.visualization.LineChart($('#chart').get(0));

          chart.draw(data, {
            title: 'Room Temperature', height: 500, is3D: true
          });

        });

      }

      // load chart lib
      google.load('visualization', '1', {
        packages: ['corechart']
      });

      // call drawChart once google charts is loaded
      google.setOnLoadCallback(drawChart);

    </script>
  </head>
  <body>
    <div id="chart" style="width: 100%;"></div>
  </body>
</html>

The above uses Google’s chart API to generate a chart from the data stored in the Sparkfun servers (thanks for Sparkfun for the code). Save the above page to a web server and browse to the page and you will see something like the following:

Room Temperature Chart

Room Temperature Chart

The chart shows the fall and rise in temperature in a room over a period of 48 hours.

Conclusion

The Electric Imp offers a simple method for connecting a device to the Electric Imp servers on the Internet. The Agent code can then pass this data on to services provided by additional third parties.

The device options used here would increase the cost of any device produced but as a proof of concept they offer a simple and convenient way of demonstrating how a device can interact with the outside world over the Internet. This example took less than 3 hours to research, build and complete – good going considering how complex this would be if this were completed in a conventional manner (Arduino, WiFly shield etc.).

Intel Galileo – First Impressions

August 21st, 2014 • Electronics, Netduino, Software Development2 Comments »

Yesterday I received my Intel Galileo rev 1 board. I know the rev 2 board is available but recently the Windows on Devices program have release the necessary firmware etc to upgrade a Galileo rev 1 board to enable it to run Windows.

See the end of this article for an update added on 6th Sept 2014.

Upgrading the Firmware

The first step was to check athe firmware and upgrade it if necessary. In my case it was necessary. Intel provide a comprehensive set of instructions on how to do this. The upgrade process took about 10 minutes.

Creating a Windows SD Card

The next step is to write Windows to a micro SD card. This step of the process took the longest to complete, about 25-30 minutes.

Booting to Windows

The next step is to verify that Windows has loaded correctly. Insert the card, power on and then waiting for 2 minutes for Windows to boot. If successful you should be able to telnet to the device. I used PTTYPortable to do this and was presented with the login request.

Testing the Board

One of the first tests I normally perform is to deploy a Blinky application to the board. Once I am happy that I can deploy applications to the board I speed up Blinky by removing any code which would cause a pause. The result should be an indicator of the performance of the board and the software. So let’s give it a go.

The Galileo board offers two methods for deploying Blinky to the board:

  1. Arduino UI
  2. Visual Studio Windows application

The first method uses the board without the Windows SD card image, the second deploys a Windows application to the SD card and runs as a Windows application.

Our starting point for the tests will be the standard Blink application:

/*
  Blink
  Turns on an LED on for one second, then off for one second, repeatedly.
 
  This example code is in the public domain.
*/
 
// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
int led = 13;

// the setup routine runs once when you press reset:
void setup()
{                
    // initialize the digital pin as an output.
    pinMode(led, OUTPUT);     
}

// the loop routine runs over and over again forever:
void loop()
{
    digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
    delay(1000);               // wait for a second
    digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
    delay(1000);               // wait for a second
}

Arduino UI

Using this application in the Arduino UI for the Galileo is simple. In fact the application is one of the samples (File -> Examples -> 01.Basics -> Blink). Loading this sketch and deploying to the Galileo starts the on board LED blinking at a steadily at 1 Hz.

Visual Studio Windows Application

Microsoft have provided a Wiring API for use in Visual Studio. This allows access to the “Arduino” hardware from a Windows application. An equivalent Windows version of the same application is:

#include "stdafx.h"
#include "arduino.h"

int _tmain(int argc, _TCHAR* argv[])
{
    return RunArduinoSketch();
}

int led = 13;  // This is the pin the LED is attached to.

void setup()
{
    pinMode(led, OUTPUT); // Configure the pin for OUTPUT so you can turn on the LED.
}

// the loop routine runs over and over again forever:
void loop()
{
    digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
    Log(L"LED OFF\n");
    delay(1000);               // wait for a second
    digitalWrite(led, HIGH);    // turn the LED on by making the voltage HIGH
    Log(L"LED ON\n");
    delay(1000);               // wait for a second
}

Much of the application looks the same as the Arduino application. The main differences are the addition of the _tmain method and the Log statement. The _tmain method acts as the entry point for the application and provides a method for running an Arduino sketch (as above) or some other program logic.

The Log statements generate debug information which is displayed in Visual Studio’s Output window.

Deploying this application to the board results in… NOTHING!

Digging a little deeper into the examples section of the web site reveals the On Board LED example:

// Main.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include "arduino.h"

int _tmain(int argc, _TCHAR* argv[])
{
    return RunArduinoSketch();
}

//This application flashes the on board LED of the Galileo board by calling GPIO functions directly in the embprpusr.dll instead of using the Arduino layer.

ULONG state = LOW; // keeps track of the state of the on-board LED

void setup()
{
    GpioSetDir(LED_BUILTIN, OUTPUT); // Sets the pin to output
}

void loop()
{
    if (HIGH == state)
    {
	    state = LOW;
	} 
    else
    {
	    state = HIGH;
	}
    GpioWrite(LED_BUILTIN, state); // Writes to the pin, setting its value either HIGH (on) or LOW (off)
    Log(L"LED %s\n", (HIGH == state ? L"ON" : L"OFF"));
    Sleep(1000);
}

Compiling and deploying this application to the board results in the steady flashing of the on board LED.

Reverting to the previous sample and hooking up an oscilloscope reveals that pin 13 is indeed being toggled at 1 Hz it is just not connected directly to the on board LED.

How Fast Can We Go?

Now to find out how fast the board will actually run. Starting with the Arduino example, remove the delay statements, recompile and deploy to the board. Doing this resulted in a square wave with a 50% duty cycle and a frequency of 221Hz. That’s right Hz, not KHz!

Removing the delay and logging statements and deploying the Windows application results in a square wave. This time a 60Hz square wave with a 30% duty cycle is displayed on the oscilloscope.

There must be something wrong. Surely this board with a 400MHz processor should run faster than this.

What About the Netduino?

The Netduino has always had one issue when compared with Arduino and other similar board. Namely it is running interpreted code which is not real time due to the nature of the .NET Microframework and the way the framework runs. I have performed similar tests and I was convinced that it was faster. Only one way to find out, deploy some code to the Netduino Plus 2. This board runs the .NET Microframework on the STM32 family of microcontrollers at 168 MHz. The equivalent code to the two examples above is:

using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware.Netduino;

namespace NetduinoPlus2
{
    public class Program
    {
        public static void Main()
        {
            OutputPort dp = new OutputPort(Pins.GPIO_PIN_D13, false);
            while (true)
            {
                dp.Write(!dp.Read());
            }
        }
    }
}

Deploying this to the Netduino PLus 2 and hooking up the oscilloscope results in a square wave with a 50% duty cycle at 17.8 KHz.

Much faster than the Galileo board.

Update – 6th Sept 2014

I have been looking through the schematic for the Galileo Rev 1 board and found that not all of the GPIO pins are connected through the CY8C9540A chip but are in fact connected directly top the Quark processor. These GPIOs should be capable of higher speeds. A quick test shows that these pins (Digital 2, 3 & 10) can all generate a 1.16 KHz square wave for an application compiled in debug mode. Compiling the same applications in release more and running the application on the Galileo increases the frequency from 1.16 KHz to 1.27 KHz.

Conclusion

I’ve only had this board for a few hours but I have deployed a few of the examples. The raw GPIO speed appears lower than the interpreted .NET Microframework equivalent. The Galileo has access to a network port with easy access to the Arduino Wiring API but then so does the .NET Microframework.

Some further investigation is required I believe.

Strictly Yorkshire Photography Exhibition

August 18th, 2014 • PhotographyComments Off

A few weeks ago I had the opportunity to add some of my photos in a local exhibition of photographs taken in Yorkshire. I don’t know what came over me but I said yes. Never done this sort of thing before but nothing ventured, nothing gained. And so on Saturday 9th and Sunday 10th August I found myself exhibiting seven photographs along with about 30 other members of the Strictly Yorkshire Photography group at the Corn Exchange in Leeds.

The group rules are simple, all photographs must be taken in Yorkshire and the photographs should not contain any offensive or illegal material.

The Exhibition

The group membership is diverse and as with all groups, some members are more active than others. The range of skills and interests are varied. My own particular fields of interest are wildlife and technology as you can see by the photographs I exhibited:

Mark Stevens Display at Yorkshire Photography Exhibition

Mark Stevens Display at Yorkshire Photography Exhibition

Spirit of Yorkshire

This image was taken at a vintage Rolls Royce and Bently gathering at Castle Howard on 1st June 2014. The weather was brilliant and I could not resist taking this image of the Spirit of Ecstasy against and almost clear blue sky.

Spirit of Yorkshire

Spirit of Yorkshire

Three Months

This photograph was taken in April of 2014 in the gardens around the York Museum.

Three Months

Three Months

This is one of the few images I have actually modified to any degree. I wanted to bring out the contrast in the wall at the back of the image whilst highlighting the right-hand headstone. This was achived by using a black and white conversion in Paintshop Pro along with HDR.

Doe at Studley Royal Park

This doe was captured early in the morning on 9th April at Studley Royal Park.

Doe at Studley Royal Park

Doe at Studley Royal Park

Sonny and Charlie

Sonny and Charlie are both resue dogs owned by a friend. I took a series of over 200 images of the two dogs when out walking in mid-April this year.

Sonny in Bluebells

Sonny in Bluebells

Sonny in Bluebells

Where’s the Ball?? WHERE’s THE BALL!!!

Where's the Ball?? WHERE'S THE BALL!!

Where’s the Ball?? WHERE’S THE BALL!!

Tornado Leaving York

Tornado is the last locomotive to be built in the UK as of the time of writing. This locomotive first steamed in 2008.

Tornado Leaving York

Tornado Leaving York

This is also an example of of an image which I manipulated. For this image I converted the image to black and white and then used this to mask off all of the original image except for Tornado itself.

Just Grabbing a Takeout For the Kids

This last image shows a Robin feeding in our garden. This was taken early in the year when the young robins had just hatched.

RobinFeeding

What Did I Learn?

I have always enjoyed photography being an enthusiastic amateur. Looking at the images that were on display gave me some ideas about perspective and angles to improve the quality of the photographs I take.

Looking back I think I over prepared and I could have made my display just as effective but with a simple change such as not using frames and just using mounts.

I felt that the day was an overall success. Would I do it again? Yes, without a doubt.

Finally, I’d just like to thank everyone who took part, a great bunch of people.