Friday 19 November 2010

Glo-toob FX in the field

I got to try out my Glo-toob FX in the field two weekends ago when went for a nice long walk and overnight camp in the Brecon Beacons.

The Glo-Toob is a cylinder measuring 70mm x 19mm and weighing in at only 34g. It takes a small, high-capacity battery. Light is provided via a set of LEDs that bounce of a clever polished metal cone at one end of the cylinder. The light spread is surprisingly uniform given the directionality of LEDs.

It's a fun little piece of kit. I'd had a hope that it would provide enough light to read by; it doesn't. Even in my fairly small one-man Scarp 1 tent you get the sort of ambient light that your mother would tell you off for trying to read by.

The Scarp has a nice hook in the centre of the roof which is perfect for hanging a small lantern like the Glo-toob.

Far be it from me to suggest the Glo-toob isn't worth having; it's definitely part of my standard kit now. It's so light you'll never notice the added weight, and it really is tiny. It's nice having an omni-directional light source on in the tent without having to faff about with a head-torch to find things. And you can leave the light on when you need to step outside for an, er, well you know.

Hopefully, the next time I get out to the hills it won't be raining and blowing a ferocious gale all night and I'll be able to take a night-time shot to show the comforting glow of this little lantern.

A night on Waun Lefrith

I visited the Brecon Beacons for my first time the weekend before last. I'd driven through there before, on my way to the north of Wales usually, but never stopped to take in a walk. I'd been itching to try out my new Scarp 1 and a few magazine articles and blog posts had planted the idea that it might be worth taking in some of the Carmarthen Fans before the weather turned too cold for the current state of my outdoor gear.

I'd meant to do a full write-up of the outing but I left it too long between walking and writing, so every time I sat down to put pen to paper (keyboard to screen?) the story came out sounding contrived.

So I'm just going to post to pictures, instead.










Friday 12 November 2010

Veeam Backup SNMP trap bug

Today I got confirmation that the baffling behaviour I was seeing in the onBackupJobCompleted SNMP trap payload is in fact an actual bug. Result!

For the benefit of others who may find this, the onBackupJobCompleted trap is sent with a payload of four objects:
  • backupJobId
  • backupJobName
  • backupJobResult
  • backupJobComment
Despite the name, the backupJobId field contains the GUID of the backup session, such as you'd get back from a Get-VBRBackupSession call.

backupJobName is exactly what you'd expect, and even backupJobComments is reasonably self-explanatory.

The bug is with the backupJobResult field, which at the moment contains the string "None" regardless of the actual job outcome.

As I said earlier, Veeam have confirmed this is a real bug that effects the initial release version of Veeam Backup and Replication v5, and will be fixed in the next release; but they haven't been forthcoming with a release date yet. So that makes this particular trap just about useless right now.

VCAP recap

Yesterday was the big day. I spent more of my day on the train into central London than I did in the exam suite, but only just! The beta exam clocked in at just under 4 hours as they threw more questions at us than you're likely to get in the shipping version of the exam.

And it was tough, too. Two weeks of cramming isn't enough time to get it all down. They're testing knowledge of a lot of subjects across technical, design, and project arenas with a broad mixture of highly conceptual and disturbingly specific questions. Not surprisingly it's all covered in the official exam blueprint, and studying the recommended reading from the blueprint will prepare you for the exam, but in much the same way that reading the entire unabridged edition of the Encyclopaedia Britannica will prepare you for the local pub quiz. There's no substitute for experience here, and my lack of experience with large design projects was a handicap. But I was expecting that.

The beta exam allowed for less than 2 minutes per question which put the pressure on, especially when there were certain classes of interactive question that demanded 5-10 minutes each. Since you have no idea what types of questions are coming, time management is very difficult, particularly towards the end. Do I rush through these multiple choice questions just in case there's a big one coming up, or take a more considered approach and risk running out of time? I went for the former and actually ended up with 8 minutes in hand at the end. It made me feel like I'd chosen wrong.

Frequently I found there wasn't enough space on the screen for the question. It was frustrating to have to answer questions against a scenario when the scenario ran off the right-hand edge of the screen. Worse were the interactive questions that didn't leave enough space on the screen to create an answer. I don't know if this is a general problem or if the testing centre I visited were just running too low a screen resolution. There were a lot of basic proof-reading and clarity of meaning issues.

I doubt it will matter though. I fully expect the exam grade will be based on a weighted sum across different subject areas and a fail in, say, the projecty bit won't make up for good performance in, say, a logical designy bit. On balance, I'm expecting a fail because of my performance on a couple of sections that I just don't have enough background in to feel confident in.

The afterward made it sound like only those who passed would be contacted once VMware had had time to collate and review all the beta test scores, so I may never know just how badly I've done :)

Tuesday 26 October 2010

VCAP DCD Exam Beta

Last week I unexpectedly received, from VMware, an invitation to sit the beta VCAP4 Datacentre Design exam. It comes at a discount rate, but with an obligation to provide feedback on the exam itself.

From the wording of the email it seems anyone with a VCP4 is eligible to sit the beta exam, so I'm not sure if I won a lottery for this or if everyone's had an invitation.

Whatever the cause, I've booked myself in for the 11th of November, and will be spending most nights between now and then with a copy of the exam blueprint and a stern resolve.

Friday 22 October 2010

New Stuff

I've made a few acquisitions in the last month.

Tomorrow morning I'm off to the post office to pick up a pack of assorted Fuizion freeze dried food sachets, which everyone seems to have been raving about.

I've ordered a Glo-Toob FX for small, lightweight in-tent lighting. I've heard good things about them and as it's always a pain to create ambient light with a head torch, which are usually very directional by design, I'm looking forward to seeing how it performs. Whether it's bright enough to read by remains to be seen, but I live in hope.

On a similar note, I've added a Petzl Tikka2 XP to my collection as well. I actually quite like my dirt cheap Energizer LED torch that I got for £5 at a Millets in Oban and have never seen sold anywhere since. But I've started cycling in to work now and the throw wouldn't be long enough to see where I was going on a dark winter's evening. I'm delighted with the effectiveness of the Petzl's diffuser, but slightly irritated by the noticeable flickering of the LEDs in economy mode.

I wish I could afford a decent insulated jacket so I can get out in the dead of winter. I'm looking at you, PHD Minimus jacket. And by decent I of course mean top of the line. Ahem.

Friday 1 October 2010

TarpTent Scarp 1: First impressions

I got my Scarp 1 yesterday, merely a week after ordering it from the US. I emailed Henry the day after I ordered it politely inquiring if I'd be lucky enough to receive a tent out of the fresh batch of Scarps that had just arrived. Henry replied the same day - he'd already shipped it!

I spent the best part of the week refreshing the USPS parcel tracking page.

Anyway, on to the unboxing. My first impressions of the tent in the bag but out of the box was that it seemed heavier, and larger than I was hoping for. It turns out this was an illusion. In a straight hand-holding test it's practically the same weight as my Eureka Spitfire 1 tent, maybe lighter. I havent put it on the scales.

The reason the size threw me is that for the last few months I've been stuffing my Spitfire into a compression sack and compressing the hell out of it until it's the size of a large grapefruit, and packing the poles and pegs separately. So last night, for comparisson's sake I repacked my Spitfire in its original bag to find it's about 20% fatter than the Scarp 1, if the same length. That's nice.

I won't be able to pack the Scarp 1 down the way I did the Spitfire because of the carbon rods at the ends that give the tent its corner structure, much like you can see in the Akto and Laser tents. You can take the rods out of course (to replace them if they break I guess) but there are 10 of them and that's a massive faff. It just means I need to think differently the next time I pack.

I had just enough space in my bedroom to set the tent up in its free-standing mode! Setup in this fashion is a breeze, and very intuitive. I'm sure the more common staking mode is just as easy. Henry provides a simple set of printed instructions and once you unfurl the tent it's easy to see how the corner and end support structures work.

This is a well thought-out tent. There are lots of venting options; one at each end, and the top vents, and the side of the fly slides up the main arch pole as well, and while I was initially put out by the lack of a double-zip on the fly doors I think it's compensated for by the presence of the clip and velcro patch at the base of the door so you can have the door unzipped the whole way but kept mostly closed by these fixing points.

The inner is suspended from the fly at a number of hanging points, but the base of the inner is floating in that it has no ground attachments of its own. It's a bit tricky (almost impossible?) to get to the end vents from inside the tent without detatching the inner first, but that's not hard.

The vestibules are quite narrow, but there are two of them, so I'm thinking that one of them can basically stay empty for cooking and whatnot, with kit stashed in the other. Also, the floating inner means it can be pegged back easily to make more room as others have already done.

I'd read about the huge amount of space inside before buying so I was expecting big things. But it's not that big. It's bigger than the Spitfire of course, and with the straight walls, more of it is usable. I guess that never having owned a Laser Comp I don't have the same perspective. Don't get me wrong, there's room, but once you start bringing clothes and gear inside your sleeping mat is going to start getting cozy with it all. But with all that vertical space it won't feel crowded.

All I need now is some silicone and a spare couple of hours to seal the seams and I can get out and use it.

So far I'm still excited about the tent, and I'm looking forward to finishing it up and getting outside with it for the first time.

Friday 10 September 2010

A lighter load, and the rest

I managed to get out on dartmoor twice the weekend before last; once on the Friday night for a quick getaway, and more seriously on the Sunday night where Jackie and I managed a 12 mile round-trip hike.

I'm excited because a few recent purchases (ThermaRest Prolite, Montane Featherlite) and a slightly more ruthless approach to packing has reduced the weight of my pack by a couple of kilograms, but the best part is that with a cheap compression sack my Eureka Spitfire tent takes up a lot less room and I can now fit back in to my much lighter 45L Osprey pack for one or two-day trips instead of the massive and heavy 65+10L Berghaus beast.

A one-man tent with a decent vestibule is still high on my wishlist so I can drop the tarp I'm currently using to make a verandah, and as Jackie's more enthusiastic about nights away than I thought she might be, I'd really like to find a larger, lighter alternative to the depressingly pokey, cheap, and disproportionately heavy Millets special two-man tent we have to lug around at the moment.

I originally had my sights set on the venerable Terra-Nova Laser Comp, but I've been put off recently by questions about its stability in high-wind, what with nothing to support the tunnel structure and only a single end-guy. But it's still one of the best weight-to-room ratio tents that (my kind of) money can buy.

For a two-man, I'm tempted by a tarptent design. But again, the poor hydrostatic-head of silicon impregnated nylon is a worry - not so much for the fly, but the floor. So you start having to lug about a footprint just to keep the floor dry.

Anyway, I need to move on the one-man tent soon. The Eureka's all-mesh inner won't cut it as the temperature drops.

Thursday 2 September 2010

More on SSD longevity

This whitepaper published by Western Digital is an interesting read and provides a couple of well thought-out equations for estimating SSD life with a known workload.

The trick is knowing your workload, and fitting it to the simple parameters in the WD LifeEST model.

Wednesday 1 September 2010

Exchange and SSDs

I'm designing an Exchange 2010 system at the moment as an upgrade from a clustered Exchange 2003 environment.

I won't go in to the details about the restrictions in the current system, save to say that even after having lots of physical spindles thrown at it, it's still IO bound.

So the new system is going to have SSD. And what a lot of fun that's turning out to be. Try finding some hard data on the life expectency of MLC SSDs under genuine enterprise (i.e. not desktop) workloads. Go on, I dare you. I'd settle for real-world life expectencies of MLC drives full stop.

Instead, everyone's throwing around the 10,000 program/erase cycle design spec with a caution to keep an eye on it, and the vendors seem reluctant to let go any real information how many writes you can expect to get away with before hard errors start to crop up.

Use SLC you say? Fair enough, but until the cost per GB comes down it's still likely to be more cost effective to burn through a couple of same-size MLCs on a price per GB per year equation. I think. See above re: vagaries of predicting drive failure.

Either SSD is still finding its feet in the enterprise space, or people just aren't writing about it yet. Or maybe the drives just haven't started to fail.

Reliability will improve as the technology continues to mature, and it's maturing fast. Even if I only get 6 months before the first drives start to burn out, which I think is exceedingly pessimistic, the replacement drives are likely to be hardier, and cheaper even then.

Anyway, I'm looking forward to getting of the design sheet on this one and starting to push bytes around. Stay tuned.

Tuesday 31 August 2010

Hello World

Hi, Sam here.

When I heard Brendan was going to be starting a blog to help with the VCAP study, I thought it would be a great opportunity for me to stop procrastinating and work on it too.

So just a brief background on me. Started using VMWare back in 2000, first production ESX site in 2003 (ESX2.5), currently running ESXi 4.0 update 2 with an upgrade to 4.1 pending in 3 weeks. We have been running effectively "cloud services" since 2005 and have offered this under the guise of Utility Computing.

In the coming months, I hope to cover quite a bit on the VCAP blueprint as it relates to my current environment. Along with that I want to cover off on a few areas that appear to be lacking from what I have heard in podcasts and user group discussions. Things like developing a cost model, Billing Models, Virtualised Storage, Monitoring (Open Source Solutions), Monitoring (COTS) and some other tit bits learned by blooding my nose on the sharp end of innovation.

So enough from me now and talk to you all soon.

Saturday 7 August 2010

Building an ESXi provisioning environment: Part 1

There are three things I knew I was going to need to build my provisioning environment:
  • A DHCP server
  • A TFTP server
  • An FTP or web server
The DHCP server should be obvious. I need my hosts to auto-configure their network awareness, and they need to know where to pick up the PXE image they're going to use to bootstrap the ESXi install process.

The TFTP server is going to provide the PXE boot image and ESXi bootstrap files.

The FTP or web server is the repository from which the ESXi installable image, kickstart script (ESXi install script), and post-install configure/build scripts are going to come from.

As far as I'm concerned, the quickest and least resource intensive way to achieve all three is to knock up a small Linux box. I chose Debian.

The DHCP server is of the ISC variety and is provided by the dhcp3-server package.

I went with the tftpd-hpa package, but any TFTP server will work for this - I'm not using any interesting features.

I opted for a web server over an FTP repository for my install media as it lends itself to a configuration script repository better than FTP. Apache's httpd server is my weapon of choice and Debian puts it in the apache2 package.

The PXE boot environment files we need come from the syslinux-common package. You'll need that too.

Install with:

vm-sys-cfg:~# apt-get install dhcp3-server tftpd-hpa apache2 syslinux-common

I've configured my host with the IP address of 10.0.0.1/24, which sets the scene for my DHCP server configuration.

My basic dhcpd configuration is:

subnet 10.0.0.0 netmask 255.255.255.0 {
range 10.0.0.201 10.0.0.210;
}


I'm not expecting to be building more than ten hosts at once, but it's easy to extend anyway.

The next part is the PXE, or in this case gPXE configuration. Now I used the ESXi 4.1 setup guide to, er, guide me on the next part as I'd never heard of gPXE before now. Unfortunately, the VMware setup guide's gPXE example is poorly explained. They spend a lot of time talking about how great gPXE is because you can boot from HTTP or FTP (or iSCSI, or ATA-over-Ethernet, or whatever) but their guide will set you up to boot from TFTP, just like a regular PXE stack, so what's the point?

The VMware guide includes:

option space gpxe;
option gpxe-encap-opts code 175 = encapsulate gpxe;
option gpxe.bus-id code 177 = string;

class "pxeclients" {
match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
next-server 10.0.0.1;
if not exists gpxe.bus-id {
filename "/srv/tftp/gpxelinux.0";
}
}


In a nutshell this defines a DHCP user class called "pxeclients" based on whether or not the vendor-class-identifier client identifier option contains the string "PXEClient" in its first nine characters. Not surprisingly, the PXE ROM in your network card does announce this.

So inside this class we can specify options and configuration that only apply to PXE clients. I've given the IP address of my Linux host as the next-server option. That's where the PXE client looks for its boot files. The next bit:

if not exists gpxe.bus-id {
filename "/gpxelinux.0";
}


basically says: if you've not identified yourself as a gPXE client, retrieve and load the /gpxelinux.0 image.

tftpd-hpa uses /var/lib/tftpboot as its root directory. Put /usr/lib/syslinux/gpxelinux.0 in there. i.e.

vm-sys-cfg:~# find /var/lib/tftpboot/
/var/lib/tftpboot/
/var/lib/tftpboot/gpxelinux.0


Cool. But what about if you are a gPXE client? Then you'll try and pull down your configuration file based on where you came from: TFTP. I want to use HTTP as my boot protocol from this point. So I've changed that stanza to read:

if not exists gpxe.bus-id {
filename "/gpxelinux.0";
} else {
filename "http://10.0.0.1/";
}


Now everything that gPXE does from this point will be relative to http://10.0.0.1/, which means I've got to set up my web server file tree.

My apache document root contains this file structure:

vm-sys-cfg:~# cd /var/www
vm-sys-cfg:/var/www# find
.
./index.html
./cfg
./cfg/cfg-00:0c:29:f2:55:a7.sh
./dist
./dist/imagedd.md5
./dist/imagedd.bz2
./ks
./ks/ks.cfg
./boot
./boot/vmkboot.gz
./boot/vmkernel.gz
./boot/ienviron.vgz
./boot/cim.vgz
./boot/sys.vgz
./boot/install.vgz
./boot/mboot.c32
./boot/menu.c32
./pxelinux.cfg
./pxelinux.cfg/default
./gpxelinux.0


The ./pxelinux.cfg/default file will be pulled down by the gPXE image. Mine's been ripped pretty much straight out of the VMware manual:

default 1
prompt 1
menu title VMware VMvisor Boot Menu
timeout 50

label 1
kernel boot/mboot.c32
append boot/vmkboot.gz ks=http://10.0.0.1/ks/ks.cfg --- boot/vmkernel.gz 
--- boot/sys.vgz --- boot/cim.vgz --- boot/ienviron.vgz --- boot/install.vgz

label 0
localboot 0x80


(Keep the append directive on a single line)

It gives me two options. I can either boot into the ESXi installer (label 1), or boot from the local disk (label 0). The default is the ESXi installer which will be invoked automatically after 50 seconds. Cool.

Everything in my ./boot directory came off the root of the ESXi install CD, as did the ./dist/imagedd.bz2 and ./dist/imagedd.md5 files. I could have just dumped everything in the root of /var/www but this way is neater.

The last part of the magic is the kickstart script referenced by the ks=http://10.0.0.1/ks/ks.cfg part of my PXE boot line.

For now, I've used:

# Accept the VMware End User License Agreement
vmaccepteula

# Set the root password for the DCUI and Tech Support Mode
rootpw p4ssw0rd

# Erase partition table on first local disk
clearpart --firstdisk=local --overwritevmfs

# Choose the first discovered disk to install onto
autopart --firstdisk --overwritevmfs

# The installation media is in the CD-ROM drive
install url http://10.0.0.1/dist/

# Set the network to DHCP on the first network adapater
network --bootproto=dhcp --device=vmnic0 --addvmportgroup=0


The install url http://10.0.0.1/dist/ directive says to perform the install from my website.

And there we go. Everything you need to automatically build an army of basic ESXi servers. The resulting ESXi servers will use DHCP for their default vmkernel adapter.

Next time I'll cover my lab DNS setup and automatically deploying a basic per-host configuration on my freshly built ESXi systems using the %firstboot kickstart section.

VCAP and building an ESXi provisioning environment

So I'm starting down the VMware VCAP road, and I want to be thorough about it, because I've read the first hand reports of people who've sat the VCAP Datacenter Administration beta exam and it sounds tough.

Being thorough means a painstaking walk through the exam blueprint. And there's going to be lots to learn as the VCAP isn't just about basic vCenter and the vSphere hypervisor any more, and that means spending time with Orchestrator, vShield Zones, the PowerCLI and much more besides; stuff which I haven't had a need to explore up to now.

But the best place for me to start is with ESXi 4.1, or as it now seems to be called (according to the download), VMvisor. Partly this is because up to now all of my VMware hypervisor work has been with ESX, console operating system and all, and that's for historical reasons (I've been doing this since before ESXi existed). But mostly it's because vSphere 4.1 is the last time we'll see the ESX service console as all future versions will be ESXi only.

Now, in the interests of making life easy for myself, I've built a simple provisioning environment to take advantage of the new scripted install mode in ESXi 4.1. This means I can have a central repository of build and configure scripts and spin up an arbitrarily complex ESXi test lab at the push of a couple of PXE-capable network-card buttons. This is a Good Thing™ because each task/step in the exam blueprint isn't necessarily going to be feature-compatible with the preceding or subsequent steps so being able to tear down and rebuild a lab repeatably and consistently is going to prove essential.

I'm building my ESXi lab on top of ESXi itself. Each 4GB physical host will house at least two 2GB ESXi systems. I'm doing that because it lets me take advantage of memory over-provisioning since I don't have the budget for more than 4GB per host, and I can have as many virtual network cards per virtual-ESXi as I need. Throw in a few Linux VMs as routers and I've got a WAN in a box as well. There'll be more about the lab architecture later.

The test lab doesn't have to be fast, it just has to work - so I don't mind so much if I end up swapping with the virtual-ESXi guests.

Anyway, that's the precis. The build environment comes next.

Monday 17 May 2010

Fa Jin (發劲)

I learned an important lesson about fa jin (發劲) on Sunday while training pao chui (炮捶 - 老架二路); don't over-reach the end-stop. It's a simple principle really, and an obvious one. Don't over-stretch the punch. Don't over-stretch an elbow, or a shoulder. Most of the 太极拳 fa jin expressions are design for use at very close range. Even strikes like that found in 掩手肱拳 (Hidden Punch) are close range attacks.

Mentally bringing my "target" two inches closer to my body made my expression of fa jin (發劲) in the punch so much sharper, and more balanced.

Pao chui (炮捶) was a lot of fun to train, but I'm not ready for it yet.

On a related note, while looking up the characterisation of 發劲 I found a wonderful short video of Chen Xiao Wang (陈小旺) demonstrating applications of the first Buddha's Warrior Attendant Pounds the Mortar (金刚捣碓).



And remember, kids; you can't 發劲 round corners.

Wednesday 21 April 2010

A heavy load to carry

I need to reduce the weight of my pack. I'm at 10Kg before I add food and water, and I'm still shy a couple of essentials.

Also: need walking poles.

Jetboil cookery

After being disappointed with the recipes on offer at the Jetboil site, I wanted to have a go at my own. These were both made in the PCS cup and will serve one person nicely. In both cases I periodically re-start the flame to reheat the Jetboil, but given how well the cozy keeps the heat in the pot this might even be redundant. Anyway, I present: risotto, and sponge with custard!

Mushroom and Sundried Tomato Risotto

Equipment:
  • Jetboil PCS cup


Ingredients:
  • 20g butter or oil
  • 100g Risotto rice
  • 5 dried porcini mushrooms
  • 5 dry sun-dried tomato pieces
  • 1 chicken stock cube


Method:

Put the butter in the bottom of the Jetboil cup and melt it over a low heat. Add the rice and fry gently for up to 1 minute.

Add water up to the maximum fill line and stir to make sure the rice isn't sticking to the bottom.

Crumble the stock cube and add it to the water. Turn up the heat a little and place the lid on the pot until the water boils. Keep an eye on it at this point as it will boil over quickly if left unattended.

Remove the lid and stir again to make sure the rice isn't sticking. Add the mushroom and sun-dried tomato pieces.

Bring back to the boil with the lid on then turn the gas off completely.

Let sit for 5 or so minutes then stir, making sure to scrape the bottom of the cup. Replace the lid and return to the boil again before once more extinguishing the flame.

Repeat this process 3 or 4 times until the water is almost completely absorbed by the rice. The rice grains should be soft but with a hint of hardness at the centre.

Add pepper to taste, and if you've got some, a teaspoon or two of lemon juice. Magic.

Steamed syrup sponge with custard

Equipment:
  • Jetboil PCS cup
  • Jetboil pot support
  • Jetboil measuring cup/fluxring cover
  • Cup or small metal ramekin


Sponge Ingredients:
  • 20g butter
  • 20g self-raising flour
  • 20g caster sugar
  • 1 Tbsp milk powder
  • 1 beaten egg
  • 1 Tbsp Golden syrup, honey, jam or whatever


Custard Ingredients:
  • 1 heaped Tbsp milk powder
  • 1 heaped Tbsp custard powder
  • 2 tsp sugar


Method:

Mix the dry ingredients together in advance and add the egg and butter on the trail. Or, if you're game, you can add the butter and/or the egg in advance too!

Put the put support, or another suitable frame (an egg ring would do,) in to the bottom of the PCS cup. The ramekin is going to sit on top of it. Add water to the PCS cup up to the top of the pot support.

Put the Golden syrup into the bottom of the ramekin.

Mix the wet and dry ingredients together in the measuring cup and spoon them in to the ramekin on top of the syrup. Lower the ramekin down on top of the pot support. If it floats, you've got too much water in your Jetboil!

Pop the lid on and light the flame. Bring the water to a gentle boil then turn off the flame. Wait 5 minutes. Turn the flame on again and bring the water back to the boil. Turn off the flame. Repeat. In 15-20 minutes you should have a nice fluffy looking sponge in your ramekin! If you resisted taking the lid off to peek, and you blocked the holes in the lid to create a pressure cooker environment, it'll cook faster.

Give it a moment to cool down, then lift the ramekin out of the PCS cup.

Now at this point I removed the pot support from the PCS cup but left the water in for making the custard, because I'm hardcore. If you don't like that idea, replace it and start again. 1 cup (250ml) of water will yield a lot of custard!

Add the milk powder, custard powder, and sugar to the water and stir vigorously to ensure it dissolves instead of clumping. Bring the water to a boil to thicken the custard.

If you turn the ramekin upside-down the sponge should pop right out, covered in gooey syrup. You can either drop it straight into the custard and eat from the pot, or use a separate bowl.

Bon appetit.