tag:blogger.com,1999:blog-88589218361210881312024-03-13T23:13:14.850-07:00Adam Horvath's blogprogramming, algorithm, and a little something elseAdamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.comBlogger81125tag:blogger.com,1999:blog-8858921836121088131.post-66135886420898819612024-01-16T02:00:00.000-08:002024-01-16T02:03:46.269-08:00Convert animated WEBP to MP4<p> Animated WebP, similar to animated GIF, is not the best video format to work with, but this is what the usual SVD (image or text to video) and ComfyUI will generate.</p><p>Unfortunately, FFMPEG does not support WebP to MP4 conversion yet, so this simple frame-by-frame conversation had to be done. As it's generating a PNG for each frame, it's rather slow, so it's only suitable for short animations.</p><p>The below simple python code converts the animated WebP into MP4, using the predefined FPS.</p><p>Don't forget to </p><p><span style="font-family: courier;">pip3 install <b>pillow</b> <b>moviepy</b></span></p><p><b><br /></b></p>
<!--HTML generated using hilite.me--><div style="background: rgb(255, 255, 255); border-color: gray; border-image: initial; border-style: solid; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;"><pre style="line-height: 125%; margin: 0px;"><span style="color: #008800; font-weight: bold;">import</span> <span style="color: #0e84b5; font-weight: bold;">os</span>
<span style="color: #008800; font-weight: bold;">import</span> <span style="color: #0e84b5; font-weight: bold;">shutil</span>
<span style="color: #008800; font-weight: bold;">import</span> <span style="color: #0e84b5; font-weight: bold;">tempfile</span>
<span style="color: #008800; font-weight: bold;">import</span> <span style="color: #0e84b5; font-weight: bold;">argparse</span>
<span style="color: #008800; font-weight: bold;">from</span> <span style="color: #0e84b5; font-weight: bold;">moviepy.video.io.ImageSequenceClip</span> <span style="color: #008800; font-weight: bold;">import</span> ImageSequenceClip
<span style="color: #008800; font-weight: bold;">import</span> <span style="color: #0e84b5; font-weight: bold;">PIL.Image</span>
<span style="color: #008800; font-weight: bold;">def</span> <span style="color: #0066bb; font-weight: bold;">analyse_image</span>(path):
im <span style="color: #333333;">=</span> PIL<span style="color: #333333;">.</span>Image<span style="color: #333333;">.</span>open(path)
results <span style="color: #333333;">=</span> {
<span style="background-color: #fff0f0;">'size'</span>: im<span style="color: #333333;">.</span>size,
<span style="background-color: #fff0f0;">'mode'</span>: <span style="background-color: #fff0f0;">'full'</span>,
}
<span style="color: #008800; font-weight: bold;">try</span>:
<span style="color: #008800; font-weight: bold;">while</span> <span style="color: #007020;">True</span>:
<span style="color: #008800; font-weight: bold;">if</span> im<span style="color: #333333;">.</span>tile:
tile <span style="color: #333333;">=</span> im<span style="color: #333333;">.</span>tile[<span style="color: #0000dd; font-weight: bold;">0</span>]
update_region <span style="color: #333333;">=</span> tile[<span style="color: #0000dd; font-weight: bold;">1</span>]
update_region_dimensions <span style="color: #333333;">=</span> update_region[<span style="color: #0000dd; font-weight: bold;">2</span>:]
<span style="color: #008800; font-weight: bold;">if</span> update_region_dimensions <span style="color: #333333;">!=</span> im<span style="color: #333333;">.</span>size:
results[<span style="background-color: #fff0f0;">'mode'</span>] <span style="color: #333333;">=</span> <span style="background-color: #fff0f0;">'partial'</span>
<span style="color: #008800; font-weight: bold;">break</span>
im<span style="color: #333333;">.</span>seek(im<span style="color: #333333;">.</span>tell() <span style="color: #333333;">+</span> <span style="color: #0000dd; font-weight: bold;">1</span>)
<span style="color: #008800; font-weight: bold;">except</span> <span style="color: red; font-weight: bold;">EOFError</span>:
<span style="color: #008800; font-weight: bold;">pass</span>
<span style="color: #008800; font-weight: bold;">return</span> results
<span style="color: #008800; font-weight: bold;">def</span> <span style="color: #0066bb; font-weight: bold;">process_image</span>(path, temp_dir):
images <span style="color: #333333;">=</span> []
mode <span style="color: #333333;">=</span> analyse_image(path)[<span style="background-color: #fff0f0;">'mode'</span>]
im <span style="color: #333333;">=</span> PIL<span style="color: #333333;">.</span>Image<span style="color: #333333;">.</span>open(path)
i <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">0</span>
p <span style="color: #333333;">=</span> im<span style="color: #333333;">.</span>getpalette()
last_frame <span style="color: #333333;">=</span> im<span style="color: #333333;">.</span>convert(<span style="background-color: #fff0f0;">'RGBA'</span>)
<span style="color: #008800; font-weight: bold;">try</span>:
<span style="color: #008800; font-weight: bold;">while</span> <span style="color: #007020;">True</span>:
basename <span style="color: #333333;">=</span> os<span style="color: #333333;">.</span>path<span style="color: #333333;">.</span>basename(path)
output_folder <span style="color: #333333;">=</span> temp_dir
frame_file_name <span style="color: #333333;">=</span> os<span style="color: #333333;">.</span>path<span style="color: #333333;">.</span>join(output_folder, f<span style="background-color: #fff0f0;">'{os.path.splitext(basename)[0]}-{i}.png'</span>)
<span style="color: #008800; font-weight: bold;">print</span>(f<span style="background-color: #fff0f0;">"saving {path} ({mode}) frame {i}, {im.size} {im.tile} to {frame_file_name}"</span>)
<span style="color: #008800; font-weight: bold;">if</span> <span style="background-color: #fff0f0;">'.gif'</span> <span style="color: black; font-weight: bold;">in</span> path:
<span style="color: #008800; font-weight: bold;">if</span> <span style="color: black; font-weight: bold;">not</span> im<span style="color: #333333;">.</span>getpalette():
im<span style="color: #333333;">.</span>putpalette(p)
new_frame <span style="color: #333333;">=</span> PIL<span style="color: #333333;">.</span>Image<span style="color: #333333;">.</span>new(<span style="background-color: #fff0f0;">'RGBA'</span>, im<span style="color: #333333;">.</span>size)
<span style="color: #008800; font-weight: bold;">if</span> mode <span style="color: #333333;">==</span> <span style="background-color: #fff0f0;">'partial'</span>:
new_frame<span style="color: #333333;">.</span>paste(last_frame)
new_frame<span style="color: #333333;">.</span>paste(im, (<span style="color: #0000dd; font-weight: bold;">0</span>, <span style="color: #0000dd; font-weight: bold;">0</span>), im<span style="color: #333333;">.</span>convert(<span style="background-color: #fff0f0;">'RGBA'</span>))
new_frame<span style="color: #333333;">.</span>save(frame_file_name, <span style="background-color: #fff0f0;">'PNG'</span>)
images<span style="color: #333333;">.</span>append(frame_file_name)
i <span style="color: #333333;">+=</span> <span style="color: #0000dd; font-weight: bold;">1</span>
last_frame <span style="color: #333333;">=</span> new_frame
im<span style="color: #333333;">.</span>seek(im<span style="color: #333333;">.</span>tell() <span style="color: #333333;">+</span> <span style="color: #0000dd; font-weight: bold;">1</span>)
<span style="color: #008800; font-weight: bold;">except</span> <span style="color: red; font-weight: bold;">EOFError</span>:
<span style="color: #008800; font-weight: bold;">pass</span>
<span style="color: #008800; font-weight: bold;">return</span> images
<span style="color: #008800; font-weight: bold;">def</span> <span style="color: #0066bb; font-weight: bold;">webp_mp4</span>(input_file, output_file<span style="color: #333333;">=</span><span style="color: #007020;">None</span>, fps<span style="color: #333333;">=</span><span style="color: #0000dd; font-weight: bold;">20</span>):
temp_dir <span style="color: #333333;">=</span> tempfile<span style="color: #333333;">.</span>mkdtemp()
<span style="color: #008800; font-weight: bold;">try</span>:
images <span style="color: #333333;">=</span> process_image(input_file, temp_dir)
<span style="color: #008800; font-weight: bold;">if</span> output_file <span style="color: black; font-weight: bold;">is</span> <span style="color: #007020;">None</span>:
output_file <span style="color: #333333;">=</span> os<span style="color: #333333;">.</span>path<span style="color: #333333;">.</span>splitext(input_file)[<span style="color: #0000dd; font-weight: bold;">0</span>] <span style="color: #333333;">+</span> <span style="background-color: #fff0f0;">'.mp4'</span>
clip <span style="color: #333333;">=</span> ImageSequenceClip(images, fps<span style="color: #333333;">=</span>fps)
clip<span style="color: #333333;">.</span>write_videofile(output_file)
<span style="color: #008800; font-weight: bold;">return</span> [output_file]
<span style="color: #008800; font-weight: bold;">finally</span>:
shutil<span style="color: #333333;">.</span>rmtree(temp_dir)
<span style="color: #008800; font-weight: bold;">def</span> <span style="color: #0066bb; font-weight: bold;">parse_arguments</span>():
parser <span style="color: #333333;">=</span> argparse<span style="color: #333333;">.</span>ArgumentParser(description<span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"Convert WEBP to MP4"</span>)
parser<span style="color: #333333;">.</span>add_argument(<span style="background-color: #fff0f0;">"input_file"</span>, help<span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"Input file name (.webp)"</span>)
parser<span style="color: #333333;">.</span>add_argument(<span style="background-color: #fff0f0;">"-o"</span>, <span style="background-color: #fff0f0;">"--output_file"</span>, help<span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"Output file name (optional)"</span>)
parser<span style="color: #333333;">.</span>add_argument(<span style="background-color: #fff0f0;">"--fps"</span>, <span style="color: #007020;">type</span><span style="color: #333333;">=</span><span style="color: #007020;">int</span>, default<span style="color: #333333;">=</span><span style="color: #0000dd; font-weight: bold;">20</span>, help<span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"Frames per second (default: 20)"</span>)
<span style="color: #008800; font-weight: bold;">return</span> parser<span style="color: #333333;">.</span>parse_args()
<span style="color: #008800; font-weight: bold;">if</span> __name__ <span style="color: #333333;">==</span> <span style="background-color: #fff0f0;">"__main__"</span>:
args <span style="color: #333333;">=</span> parse_arguments()
webp_mp4(args<span style="color: #333333;">.</span>input_file, args<span style="color: #333333;">.</span>output_file, args<span style="color: #333333;">.</span>fps)
</pre></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-88105685546329766262021-08-15T22:03:00.006-07:002021-08-15T22:06:03.298-07:00Octoprint as a systemd service - running it with high priority<p><a href="https://octoprint.org/" rel="nofollow" target="_blank"></a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5NVhbzqH1yJWjaFBD66cL_CuI2qYWZFx8m7bI3PDIzB5IDCiaR1u22nkV7dNZ6L4aSidnihDLeQ9DdIT_CsAQLKY83FfR_yG9q911zbPKHNJ5WMi-o_NqPTL-YdgaeTqOW_Ly_OCjGU8/s600/ender-3-6_1_300x%25402x.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="600" data-original-width="600" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5NVhbzqH1yJWjaFBD66cL_CuI2qYWZFx8m7bI3PDIzB5IDCiaR1u22nkV7dNZ6L4aSidnihDLeQ9DdIT_CsAQLKY83FfR_yG9q911zbPKHNJ5WMi-o_NqPTL-YdgaeTqOW_Ly_OCjGU8/w200-h200/ender-3-6_1_300x%25402x.jpg" width="200" /></a></div><br />Octoprint provides remote access to your 3D printer, and as it's written in Python, you can run it on practically any operating system.<p></p><p>If you are using a standard Raspberry PI image (for instance, because you run it on an <a href="http://www.orangepi.org/" rel="nofollow" target="_blank">Orange PI</a>), rather than the customised Octoprint image, you can manually install it and create a <b>systemd</b> service to run it:</p><p><br /></p><p><br /></p><p><br /></p><p>Add this to <span style="font-family: courier;">/etc/systemd/system/octoprint.service </span></p>
<pre><br /></pre><pre>[Unit]
Description=Octoprint
After=network.target
[Service]
ExecStart=/home/pi/octoprint/bin/octoprint serve
WorkingDirectory=/home/pi/octoprint
StandardOutput=inherit
StandardError=inherit
Restart=always
User=pi
CPUSchedulingPolicy=rr
CPUSchedulingPriority=80
[Install]
WantedBy=multi-user.target
</pre>
<p><br /></p><p>Then run</p><p><br /></p><p>
</p><pre>sudo systemctl daemon-reload
sudo systemctl enable octoprint
</pre>
<p><br /></p><p><br /></p><p>This will start Octoprint on every reboot with a high priority (so unlikely to stutter during printing).</p><p><br /></p><p><br /></p><p></p>Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-67971747607918753342021-07-08T20:24:00.003-07:002022-06-06T16:44:06.472-07:00Fusion 360 - part list dimensions, counts, and total amounts (lengths)<p> <img height="440" src="https://github.com/adam-ah/fusion360_parts_list/raw/main/fusion360_part_list_screenshot.png" width="640" /></p><p>A requirement that comes up often, especially in woodworking, is to be able to list out all materials used in bodies and components - dimensions, counts of each cut, and total lengths of materials used.</p><p><br /></p><p>There is a way to create a separate component from each body and manually add a description that will appear in a part list on a drawing, but it's unnecessarily labour intensive.</p><p><br /></p><h2 style="text-align: left;"><b>Automated parts-list addon</b></h2><p>Fusion 360 provides a <a href="https://help.autodesk.com/view/fusion360/ENU/?guid=GUID-7B5A90C8-E94C-48DA-B16B-430729B734DC" rel="nofollow" target="_blank">Python API</a> that enables direct access to the objects in a design, so I created a simple<a href="https://github.com/adam-ah/fusion360_parts_list"> Fusion 360 script</a> that can be used as an addon or run directly to display to cuts required for the design and the total amount of length used by each material type.</p><p><br /></p><p>To run directly, <a href="https://raw.githubusercontent.com/adam-ah/fusion360_parts_list/main/fusion360_part_list.py" target="_blank">download the script from GitHub</a>, place it anywhere. Then in Fusion 360 click "View" > "Show Text Commands"; in the commands window on the bottom, change from "Txt" to "Py", and run the following command:</p><p><br /></p><p><span style="font-family: courier;">import neu_dev; neu_dev.run_script("/Users/name/Downloads/fusion360_part_list.py")</span></p><p><br /></p><p>Alternatively, you can add the script as an addon, by switching to Tools, clicking "Add-Ins" / "Scripts and Add-Ins", then "Create". Simply paste in the script and click "Run".</p><p><br /></p><h2 style="text-align: left;">Sample report</h2><p><br /></p><table border="0" cellpadding="0" cellspacing="2" style="margin: 0px;"><tbody><tr><td><p style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">Component</span></p></td><td><p style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">Body</span></p></td><td><p align="right" style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">x (mm)</span></p></td><td><p align="right" style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">y (mm)</span></p></td><td><p align="right" style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">z (mm)</span></p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">rib2</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">445.00</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">rib3</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">445.00</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">vertical4</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">270.00</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">end2</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">262.50</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">end1</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">262.50</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">vertical3</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">220.00</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">vertical2</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">220.00</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">weight_holder v9</p></td><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">vertical1</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">35.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">170.00</p></td></tr></tbody></table><h2 style="margin: 16px 0px 12px;"><span style="font-size: x-large;">Total counts</span></h2><table border="0" cellpadding="0" cellspacing="2" style="margin: 0px;"><tbody><tr><td><p style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">Dimensions</span></p></td><td><p align="right" style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">Count</span></p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">35.00 x 70.00 x 170.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">1</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">35.00 x 70.00 x 220.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">2</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">35.00 x 70.00 x 262.50</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">2</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">35.00 x 70.00 x 270.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">1</p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">35.00 x 70.00 x 445.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">2</p></td></tr></tbody></table><h2 style="margin: 16px 0px 12px;"><span style="font-size: x-large;">Total lengths</span></h2><table border="0" cellpadding="0" cellspacing="2" style="margin: 0px;"><tbody><tr><td><p style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">Dimensions (mm)</span></p></td><td><p align="right" style="margin: 0px; white-space: pre-wrap;"><span style="font-weight: 600;">Total length (mm)</span></p></td></tr><tr><td style="padding-right: 15px;"><p style="margin: 0px; white-space: pre-wrap;">35.00 x 70.00</p></td><td style="padding-left: 15px;"><p align="right" style="margin: 0px; white-space: pre-wrap;">2295.0</p></td></tr></tbody></table><p style="margin: 0px; white-space: pre-wrap;"><br /><br /><span style="font-style: italic;">Note: 2 hidden bodies excluded from the lists</span></p><p><br /><br /></p><p><br /></p><p><br /></p>Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-68642848211487481852021-03-10T21:32:00.005-08:002021-03-10T21:51:55.349-08:00Triggering an Alexa routine from the command line or an Arduino compatible device (ESP32)<p>
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxdjxnv-sesfKJrVWzinwAZhmhwdQ1sI367nA2_V37CsXWa4X8l0r7_Q3yeGE1_YItXpC-5yTysU0n-ZVDMsNoY0g5vCAJXGEsq_AsMntUBw2L5845i7t7Ci5lCcma6aXqwlISYWRd11M/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="435" data-original-width="685" height="406" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxdjxnv-sesfKJrVWzinwAZhmhwdQ1sI367nA2_V37CsXWa4X8l0r7_Q3yeGE1_YItXpC-5yTysU0n-ZVDMsNoY0g5vCAJXGEsq_AsMntUBw2L5845i7t7Ci5lCcma6aXqwlISYWRd11M/w640-h406/image.png" width="640" /></a></div><br />By creating an auto-resetting virtual sensor on SmartThings.com, Alexa can
execute a routine when the virtual sensor's state is triggered via a REST call to SmartThings.com<p></p>
<h2>
<span data-preserver-spaces="true">Steps to follow</span></h2>
<h3>
<span data-preserver-spaces="true">Creating a virtual, auto-resetting sensor on SmartThings.com</span></h3>
<ul>
<li>
<span data-preserver-spaces="true">Register an account with SmartThings.com</span></li>
<li>
<span data-preserver-spaces="true">Go to <a href="https://graph.api.smartthings.com/ide/devices" rel="nofollow" target="_blank">My device
handlers</a></span></li>
<li>
<span data-preserver-spaces="true">Create a new device handler </span></li>
<li>
<span data-preserver-spaces="true">Switch to "From code" and paste in the <a href="https://github.com/adam-ah/arduino_alexa_routine/blob/main/virtual_sensor.smartthings" rel="nofollow" target="_blank">auto-resetting sensor code from here.</a></span></li>
<li>
<span data-preserver-spaces="true">Go to <a href="https://graph.api.smartthings.com/location/list" rel="nofollow" target="_blank">My Locations</a>
and add a new location (e.g., "Living Room")</span></li>
<li>
<span data-preserver-spaces="true">Go to <a href="https://graph.api.smartthings.com/device/list" rel="nofollow" target="_blank">My Devices</a> and
add a new device</span></li>
<ul>
<li class="ql-indent-1">
<span data-preserver-spaces="true">Name, Label, Zigbee Id, Device Network Id = MyVirtualButton</span></li>
<li class="ql-indent-1">
<span data-preserver-spaces="true">Type = Simulated Alexa Button (probably the last item in the
dropdown)</span></li>
<li class="ql-indent-1">
<span data-preserver-spaces="true">Location = Living Room (or whatever you created)</span></li>
<li class="ql-indent-1">
<span data-preserver-spaces="true">Hub, Group = (empty)</span></li>
</ul>
<li>
<span data-preserver-spaces="true">Once
you created the device, click on it and make sure to take a note of the GUID in the URL, which will look
something like this (the last part, c00... is the GUID of the device):
/device/show/<i>c00ab333-71e6-4a62-9c39-5df269cddd95</i></span></li></ul>
<h3>
<span data-preserver-spaces="true">Create a personal access token for SmartThings.com</span></h3>
<ul>
<li>
<span data-preserver-spaces="true">Visit
the <a href="https://smartthings.developer.samsung.com/docs/auth-and-permissions.html" rel="nofollow" target="_blank">SmartThings permissions page</a> and click the link to get a personal access
token</span></li>
<li>
<span data-preserver-spaces="true">Take
a note of this personal access token</span></li></ul>
<h3>
<span data-preserver-spaces="true">Making
the first call to SmartThings.com from the command line</span></h3>
<p>
<span data-preserver-spaces="true">To
make sure everything works so far, execute the following curl command (substitute your GUIDs):</span></p>
<p>
</p><pre>curl -H 'Authorization: Bearer: YOUR_PERSONAL_ACCESS_TOKEN' \
-d '[{"component":"main","capability":"switch","command":"on"}]' \
https://api.smartthings.com/v1/devices/YOUR_DEVICE_GUID/commands</pre>
<p>
<span data-preserver-spaces="true">Then
something like this should come back:</span></p>
<pre>{"results":[{"id":"fffb05cb-16fd-4c0c-a1de-24ab0e96b0ba","status":"ACCEPTED"}]}</pre>
<h3>
<span data-preserver-spaces="true">Setting up Alexa</span></h3>
<ul>
<li>
<span data-preserver-spaces="true">Install
the SmartThings skills on your Alexa account</span></li>
<li>
<span data-preserver-spaces="true">Discover
new devices - this should find your new MyVirtualButton sensor (note: we had to create a sensor rather than
a simple switch: Alexa cannot trigger routines based on switches, only based on sensors).</span></li>
<li>
<span data-preserver-spaces="true">Create
a new Alexa routine that is triggered by MyVirtualButton: "<i>When MyVirtualButton is...</i>"
"<i>Open</i>"</span></li></ul>
<h3>
<span data-preserver-spaces="true">Triggering
the virtual sensor from your Arduino device</span></h3>
<p>
<span data-preserver-spaces="true">The
last step is to write a simple HTTP POST call -replacing the curl command above- in C to call SmartThings.com
with the right personal token and device id. <a href="https://github.com/adam-ah/arduino_alexa_routine/blob/main/trigger_smartthings.c" rel="nofollow" target="_blank">The complete Arduino code to trigger an Alexa routine can be found here</a>.</span></p>
<p>
<br /></p>
<p>
<br /></p>
<p>
</p><p></p>Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-12427338826894288462021-03-06T14:45:00.008-08:002021-03-06T14:45:52.229-08:00 NumPy / Sklearn performance comparison - macOS, Ubuntu, Windows 10 - use Anaconda<p>One of the advantages to use a Hackintosh is that you can try the same software on the same hardware across three operating systems: how does NumPy - and therefore Sklearn - perform across different operating systems and mathematics libraries?</p><p><br /></p><p>I created a simple <a href="https://github.com/adam-ah/numpy_perftest" target="_blank">Sklearn routine</a>, which utilises a single thread for its calculations.</p><p><br /></p><h2 style="text-align: left;">Results</h2>
<table border="1">
<thead>
<tr style="font-weight: bold;">
<td>Hardware</td>
<td>Os</td>
<td>Python</td>
<td>Math library</td>
<td>Comment</td>
<td>Time (s)</td>
</tr>
</thead>
<tbody><tr>
<td>Macbook Air m1</td>
<td>MacOS Big Sur</td>
<td>3.9.1</td>
<td>CBLAS/LAPACK</td>
<td>Anaconda install</td>
<td style="text-align: right;">22</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>Ubuntu 20.04</td>
<td>3.8.5</td>
<td>OpenBLAS</td>
<td>standard apt and pip install</td>
<td style="text-align: right;">27</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>Ubuntu 20.04</td>
<td>3.9.0</td>
<td>OpenBLAS</td>
<td>standard apt and pip install</td>
<td style="text-align: right;">27</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>MacOS Catalina</td>
<td>3.8.5</td>
<td>MKL</td>
<td>Hackintosh with Clover, <br />Anaconda install</td>
<td style="text-align: right;">28</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>Windows 10</td>
<td>3.8.5</td>
<td>MKL</td>
<td>Anaconda install</td>
<td style="text-align: right;">28</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>MacOS Big Sur</td>
<td>3.9.1</td>
<td>OpenBLAS</td>
<td>Hackintosh with OpenCore, <br />brew and pip install</td>
<td style="text-align: right;">29</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>MacOS Big Sur</td>
<td>3.8.7</td>
<td>OpenBLAS</td>
<td>Hackintosh with OpenCore, <br />brew and pip install</td>
<td style="text-align: right;">32</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>MacOS Catalina</td>
<td>3.8.7</td>
<td>OpenBLAS</td>
<td>Hackintosh with Clover, <br />brew and pip install</td>
<td style="text-align: right;">33</td>
</tr>
<tr>
<td>Intel 9600k</td>
<td>Ubuntu 20.04</td>
<td>3.7.9</td>
<td>MKL</td>
<td>Anaconda Intel proprietary <br />Python+Numpy</td>
<td style="text-align: right;">34</td>
</tr>
<tr>
<td>Macbook Pro i5-7360U</td>
<td>MacOS Catalina</td>
<td>3.8.5</td>
<td>MKL</td>
<td>Anaconda install</td>
<td style="text-align: right;">38</td>
</tr>
<tr>
<td>Intel NUC N8280</td>
<td>Ubuntu 20.04</td>
<td>3.8.5</td>
<td>OpenBLAS</td>
<td>standard apt and pip install</td>
<td style="text-align: right;">181</td>
</tr>
<tr>
<td>Raspberry Pi 4</td>
<td>Raspbian Linux 10</td>
<td>3.7.3</td>
<td></td>
<td></td>
<td style="text-align: right;">181</td>
</tr>
</tbody></table>
<p><br /></p><h3 style="text-align: left;">Visually, displaying only the same hardware (9600k)</h3><p><br /></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBEr3oVBthGB1V-Rb-kCAwdWi4mKSTOW3TEGdjvPS03ukixs-mX3jKMJAzMHa72_YdkYHs5lmeF1eURjdGJ78qJwmGrHF1o2Z-lJ-KxJMleVx4t3eqFeyA64pJDbyedb9SfFBKxWlLLOI/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="783" data-original-width="831" height="603" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBEr3oVBthGB1V-Rb-kCAwdWi4mKSTOW3TEGdjvPS03ukixs-mX3jKMJAzMHa72_YdkYHs5lmeF1eURjdGJ78qJwmGrHF1o2Z-lJ-KxJMleVx4t3eqFeyA64pJDbyedb9SfFBKxWlLLOI/w640-h603/image.png" width="640" /></a></div><br /><br /><p></p><p><br /></p><h2 style="text-align: left;">Findings</h2><p><br /></p><p>There were three surprising findings:</p><p>First, the way Python and NumPy are installed does matter a lot: Anaconda's libraries are optimised and will give you an obvious performance edge compared to brew install. Install these using Anaconda instead of brew on macOS.</p><p>Second, setting up the development environment on Ubuntu was by far the easiest. Not only apt installed everything right away (no need for brew on Anaconda), the installed libraries were highly optimised and gave the extra performance that on macOS only the Anaconda install provided.</p><p>Third, the Apple M1 is really fast, even though it was fiddly to install NumPy - requires Anaconda, brew does not work. </p><p><br /></p><h2 style="text-align: left;">Takeaway message</h2><p><br /></p><p>Use the Anaconda installer - it is not only convenient, it is also highly optimised. Brew on macOS will not give you a good machine learning performance.</p>Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-46347404611132071032021-01-13T17:19:00.002-08:002021-01-13T17:19:38.013-08:00ESP32 - send a push notification from the Arduino ESP32 device to your phone<p style="text-align: left;">Sending a push notification to your phone from an event from an ESP32 is simple.</p><h1 style="text-align: left;">What you'll need (if you are using a Mac for development):</h1><p><a href="https://www.arduino.cc/en/software" rel="nofollow" target="_blank">Arduino IDE</a></p><p><a href="https://github.com/espressif/arduino-esp32/blob/master/docs/arduino-ide/mac.md" rel="nofollow" target="_blank">Arduino for ESP32</a></p><p><a href="https://www.silabs.com/developers/usb-to-uart-bridge-vcp-drivers" rel="nofollow" target="_blank">Serial port simulator for ESP32</a> to upload your code to the board</p><p>Wifi network</p><p><a href="https://ifttt.com/" rel="nofollow" target="_blank">IFTTT </a>account</p><p>IFTTT app on your phone</p><p>Wifi network that your ESP32 board can access</p><p><br /></p><h1 style="text-align: left;">Architecture</h1><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPy2EQPHmb0sRkvsA0YCPnwOs8FEkWYeMB5xsxFhJSo_HjHJggL54jXJz2hGOPtqiupp-FYC_CSQt8lf4teI12v2YmLPsQym10Elm5KsvkV9wJ6aNiaradJAN0-82q5K7I8XnYwoR-kc8/s451/push_architecture.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="388" data-original-width="451" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPy2EQPHmb0sRkvsA0YCPnwOs8FEkWYeMB5xsxFhJSo_HjHJggL54jXJz2hGOPtqiupp-FYC_CSQt8lf4teI12v2YmLPsQym10Elm5KsvkV9wJ6aNiaradJAN0-82q5K7I8XnYwoR-kc8/s16000/push_architecture.png" /></a></div><br /><h1 style="text-align: left;">Create an IFTTT applet</h1><p>This should read something like this:</p><p><i>If Maker Event "notification", then Send a notification from the IFTTT app</i></p><h1 style="text-align: left;">Change the notification template</h1><div>Go to the settings of your new IFTTT applet and change the template to include "Value1", like this:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3Y_mSAZW7c53qGJwbjTNkFGBOXF_ix4wsNVuP3zWSKz3MgFH2pFYIGX3Vmhr3P19I2T3GL9lCeFVPMZi4lixLmkzHX6FhNe4gW0osJBrA8aD5g6fsjTu2rsOONGmP_BbWkZLlNQj-kPM/s1075/Screen+Shot+2021-01-14+at+12.07.32+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1075" data-original-width="391" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3Y_mSAZW7c53qGJwbjTNkFGBOXF_ix4wsNVuP3zWSKz3MgFH2pFYIGX3Vmhr3P19I2T3GL9lCeFVPMZi4lixLmkzHX6FhNe4gW0osJBrA8aD5g6fsjTu2rsOONGmP_BbWkZLlNQj-kPM/s16000/Screen+Shot+2021-01-14+at+12.07.32+pm.png" /></a></div><br /><div><br /></div><h1 style="text-align: left;">Get your Webhook endpoint in IFTTT</h1><p>On your applet, click the triangle looking thing (webhook), then Settings, or simply go to this URL <a href="https://ifttt.com/maker_webhooks/settings">https://ifttt.com/maker_webhooks/settings</a> </p><p>Visit the private URL in your browser, replace the trigger name with "notification", so it reads like</p><p><span style="background-color: #f3f3f3; color: #333333; font-family: Menlo, "Deja Vu Sans Mono", "Bitstream Vera Sans Mono", Monaco, monospace; font-size: 14px;">...trigger/notification/</span><span style="background-color: #f3f3f3; color: #333333; font-family: Menlo, "Deja Vu Sans Mono", "Bitstream Vera Sans Mono", Monaco, monospace; font-size: 14px;">with/...</span></p><p>and in the <i>Value1</i> field type in a test message. Click "Test it" and your phone should receive the push.</p><h1 style="text-align: left;">Code to put on your ESP32 - Arduino</h1><p>The following code will send a POST request to IFTTT to trigger a message. Replace the values from the previous settings page.</p><p><br /></p>
<pre>#include <WiFi.h>
#include <HTTPClient.h>
const char* ssid = "YOUR WIFI NETWORK NAME";
const char* password = "YOUR WIFI PASSWORD";
//Your IFTTT webhook private address
const char* serverName = "https://maker.ifttt.com/trigger/notification/with/key/CHANGE_THIS_IS_THE_KEY";
const int BUTTON_PIN = 0;
const int LED_PIN = 2;
void setup() {
Serial.begin(115200);
WiFi.begin(ssid, password);
Serial.println("Connecting");
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.print("Connected to WiFi network with IP Address: ");
Serial.println(WiFi.localIP());
}
void loop() {
//Send an HTTP POST request on button press
if (WiFi.status() == WL_CONNECTED) {
if (digitalRead(BUTTON_PIN) == LOW)
{
while (digitalRead(BUTTON_PIN) == LOW)
; // Wait for button to be released
HTTPClient http;
http.begin(serverName);
http.addHeader("Content-Type", "application/json");
String httpRequestData = "{\"value1\":\"YOUR MESSAGE GOES HERE\"}";
int httpResponseCode = http.POST(httpRequestData);
Serial.print("HTTP Response code: ");
Serial.println(httpResponseCode);
http.end();
}
}
else {
Serial.println("WiFi Disconnected");
delay(5000);
}
lastTime = millis();
delay(100);
}
</pre>Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-85466122689211950402020-01-12T19:11:00.000-08:002020-04-11T02:24:19.649-07:00Building a hackintosh - Intel 9600kf, Gigabyte Aorus Z390 Pro, Radeon RX590<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Picking the most compatible hardware is essential for a Hackintosh, so I went with an Intel 9600kf CPU, a Gigabyte Aorus Z390 PRO motherboard, and a Radeon RX590 graphics card, and a 970 Evo Plus SSD.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The onboard Intel ethernet works with a custom driver (kext), and WiFi and Bluetooth work with an extra Fenvi T919 card installed.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h2 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Steps - high level</strong></h2>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<ol style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Download Catalina Installer from MacOS store on any real Mac</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Create a USB installer of Catalina (you need a 16Gb or larger USB drive for Catalina)</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Find the custom drivers you need (the kexts)</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Change BIOS settings on the new hardware</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Patch the USB installer with Clover so it can boot on a non-Mac hardware</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Install it on the new hardware</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Install Clover on the new hardware so it can boot</span></li>
<li style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; list-style-type: decimal; margin-bottom: 0pt; margin-top: 0pt;"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Fix potential USB issues by disabling some of the unused USB ports using a custom generated kext from <a href="https://github.com/corpnewt/USBMap" rel="nofollow" target="_blank">USBMap</a> </span></li>
</ol>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h2 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Steps - details</strong></h2>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Read the <a href="https://hackintosh.gitbook.io/-r-hackintosh-vanilla-desktop-guide/" rel="nofollow" target="_blank">Vanilla Guide </a>before starting</span><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">. It's not too long, but you need to understand every step. Make sure you understand NOT to install Clover on a real mac - only on a USB drive and on a Hackintosh.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">USB installer</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Download Catalina Installer from MacOS store on any real Mac and put it on the USB drive using the Vanilla guide.</span><br />
<h3 style="text-align: left;">
Find the custom drivers you need (the kexts)</h3>
</div>
<ul style="text-align: left;">
<li>To enable onboard audio, get <a href="https://github.com/acidanthera/applealc/releases" rel="nofollow" target="_blank">AppleALC.kext</a> </li>
<li>To enable onboard ethernet, get <a href="https://github.com/acidanthera/IntelMausi/releases" rel="nofollow" target="_blank">IntelMausiEthernet.kext</a> </li>
<li>To enable loading kexts, get <a href="https://github.com/acidanthera/lilu/releases" rel="nofollow" target="_blank">Lilu.kext </a></li>
<li>To temporarily enable all USB ports, use <a href="https://github.com/RehabMan/OS-X-USB-Inject-All" rel="nofollow" target="_blank">USBInjectAll.kext</a> (but fix it later - see below)</li>
<li>To enable booting on non-Mac hardware, get <a href="https://github.com/acidanthera/VirtualSMC/releases" rel="nofollow" target="_blank">VirtualSMC.kext</a></li>
<ul>
<li>From this package, also get SMCProcessor.kext and SMCSuperIO.kext for CPU fan speed and power management</li>
</ul>
<li>To enable the GPU, get <a href="https://github.com/acidanthera/whatevergreen/releases" rel="nofollow" target="_blank">WhateverGreen.kext</a> </li>
</ul>
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Change BIOS settings on the new hardware</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The most important setting is XHCI Hand-off : Enabled.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Patch the USB installer with Clover so it can boot on a non-Mac hardware</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Get Clover and Clover configurator http://mackie100projects.altervista.org/download-clover-configurator/. </span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Get <a href="https://github.com/acidanthera/macserial" rel="nofollow" target="_blank">macserial</a> to generate a valid but unused serial number </span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Use the Vanilla guide to select the right settings in Clover, but make sure to install EmuVariableUefi driver as well.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Load the Coffee Lake defaults from the Vanilla guide, then follow <a href="https://github.com/cmer/gigabyte-z390-aorus-master-hackintosh/blob/master/STEP_BY_STEP.md" rel="nofollow" target="_blank">cmer's step-by-step</a></span><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> to customise it to your settings.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Once the setup is done, copy the kexts to <span style="font-family: "courier new" , "courier" , monospace;">/EFI/CLOVER/kexts/Other</span> and the customised <span style="font-family: "courier new" , "courier" , monospace;">Config.plist</span> to <span style="font-family: "courier new" , "courier" , monospace;">/EFI/CLOVER/</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Install it on the new hardware</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Boot from the USB. If you get a corrupted screen with a prohibited sign, it's probably the USB port getting ejected during install. Plug the key into the USB hub of the motherboard or an external USB 2.0 hub and try again.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Dual boot setup</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The easiest is to pre-allocate the space for Windows during MacOS install. During install, use the advanced setup and create two MacOS partitions - the second one you can name "Free" and choose another filesystem for it. During the Windows install later, select this partition as a target, and reformat it to NTFS.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Install Clover on the new hardware so it can boot</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">After the installation, you have to install Clover with the same settings to the Hackintosh. Copy your EFIs and Config.plist to your EFI partition.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The easiest way to mount the EFI partition is to use Clover Configurator, but you can do it from the command line as well using the </span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">diskutil list</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">sudo diskutil mount /dev/diskXsY</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">command.</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h2 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Post-install fixes</strong></h2>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Check the system log</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">There are many reasons you would want to see what was happening that resulted in an error.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">sudo pmset -g log </span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Fix potential USB issues by disabling some of the unused USB ports using a custom generated kext from USBMap </strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The USB Inject kext injects all USB ports, even though the limit is 15. The downside is that all of the ports are slow ports (HS, high-speed, USB 2.0) instead of fast ports (SS, superspeed, USB 3.0).</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Each physical superspeed port is a high-speed port as well, so a physical port uses two USB slots from the available 15.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">To fix this, get USBMap and any USB device, and plug it into all the available ports you want to use. The rest can be disabled. </span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">However, watch out, one internal USB header is required for Fenvi T919 is you want to use its Bluetooth feature (plug the little cable into this and the card).</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Cmer created a <a href="https://www.youtube.com/watch?v=j3V7szXZZTc" rel="nofollow" target="_blank">great video explaining how to use the USBMap</a> tool, follow this.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">For me, I kept the following ports:</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS10 - front 1</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS09 - front 2</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS04 - back</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS03 - back</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS05 - back</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS06 - back, USB-C</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS13 - back hub ports</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS11 - internal USB connector 1 (for Bluetooth on Fenvi T919)</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS12 - internal USB connector 2</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">And disabled the followings:</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS08 - back - disable (under ethernet)</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">HS07 - back - disable (under ethernet)</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">NVRAM native support</strong></h3>
<div>
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></strong></div>
<div>
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">UPDATE: this may prevent iMessage from working!</span><br />
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></span>
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Instead of emulated NVRAM using the EmuVariableUefi.efi driver, it is possible on the Z390 to enable native NVRAM support.</span></div>
<div>
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Simply delete the EmuVariableUefi.efi from </span>EFI/CLOVER/drivers/UEFI fodler, and place the <a href="https://www.reddit.com/r/hackintosh/comments/erd2th/nvram_for_all_300_series_users_rejoice/">SSDT-PMC.aml</a> file into the EFI/CLOVER/ACPI/patched. </div>
<div>
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></strong></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Time on Windows is off</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></strong></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Windows and Mac use different timezones in the BIOS, so we have to force Windows to use the BIOS clock as UTC; otherwise, the two OSes keep displaying the wrong time and setting and resetting the clock:</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">Reg add HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation /v RealTimeIsUniversal /t REG_QWORD /d 1</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h3 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Mouse on MacOS is too slow</strong></h3>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></strong></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The fastest mouse "scaling", or speed is 3 on Catalina if you set it from the GUI. You can make it faster by executing the following commands:</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">defaults read -g com.apple.mouse.scaling </span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">defaults write -g com.apple.mouse.scaling 4</span></span></div>
<div style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="color: #1c1e29;">
<br /></div>
<h3 style="color: #1c1e29; text-align: left;">
Prevent hibernate to disk, sleep to memory only</h3>
<div style="color: #1c1e29;">
<br /></div>
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace;">sudo pmset -a hibernatemode 0</span></div>
<div style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="color: #1c1e29;">
<br /></div>
<h3 style="color: #1c1e29; text-align: left;">
Check logs if something is not right</h3>
<div style="color: #1c1e29;">
<br /></div>
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace;">sudo pmset -g log q | grep "XXX"</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<div style="text-align: left;">
<span style="color: #1c1e29; font-family: inherit;">Kernel panic log is saved to</span></div>
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace;">/Library/Logs/DiagnosticReports</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace;"><br /></span>
<h3 style="text-align: left;">
The hard drive, as a UEFI boot option is gone, the system does not boot at all</h3>
<div style="text-align: left;">
<span style="color: #1c1e29; font-weight: normal;">You probably lost the UEFI setting from the NVRAM of the bios, so you need to re-add your Clover efi file as a boot option, using the <a href="https://wiki.gentoo.org/wiki/Efibootmgr" rel="nofollow">efibootmgr</a> command in Linux.</span></div>
<div style="text-align: left;">
<span style="color: #1c1e29; font-weight: normal;"><br /></span></div>
<div style="text-align: left;">
<span style="color: #1c1e29; font-weight: normal;">In my case, as it is an NVME drive, I had to execute the following command as root:</span></div>
<div style="text-align: left;">
<span style="color: #1c1e29; font-weight: normal;"><br /></span></div>
<div style="text-align: left;">
<span style="color: #1c1e29; font-weight: normal;"><span style="font-family: Courier New, Courier, monospace;">efibootmgr -c -d /dev/nvme0n1 -p 1 -L "BOOT" -l '\EFI\BOOT\BOOTX64.efi'</span></span></div>
<div style="text-align: left;">
<span style="color: #1c1e29; font-weight: normal;"><br /></span></div>
<h3 style="text-align: left;">
<span style="color: #1c1e29;">My config.plist</span></h3>
</div>
<div style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="color: #1c1e29;">Don't paste this in - it's just a readable text format, not .plist format.</span><br />
<span style="color: #1c1e29;"><br /></span>
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;">{</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ACPI = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> DSDT = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Fixes = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> AddMCHC = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixHPET = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixIPIC = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixRTC = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixShutdown = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixTMR = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Patches = (</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Comment = "change XHCI to XHC";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Disabled = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Find = {length = 4, bytes = 0x58484349};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Replace = {length = 4, bytes = 0x5848435f};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Comment = "change XHC1 to XHC";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Disabled = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Find = {length = 4, bytes = 0x58484331};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Replace = {length = 4, bytes = 0x5848435f};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Comment = "change SAT0 to SATA";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Disabled = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Find = {length = 4, bytes = 0x53415430};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Replace = {length = 4, bytes = 0x53415441};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Comment = "Fix 300-series RTC Bug";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Disabled = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Find = {length = 8, bytes = 0xa00a935354415301};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Replace = {length = 8, bytes = 0xa00a910aff0bffff};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Comment = "change HECI to IMEI";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Disabled = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Find = {length = 4, bytes = 0x48454349};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Replace = {length = 4, bytes = 0x494d4549};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> );</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> DropTables = (</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Signature = DMAR;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Signature = MATS;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> );</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixHeaders = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> HaltEnabler = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> SSDT = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Generate = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> PluginType = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Boot = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Arguments = "keepsyms=1 dart=0 slide=0 darkwake=8 shikigva=32 alcid=11";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> DefaultVolume = MacOS;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Timeout = 5;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> XMPDetection = Yes;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Devices = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Audio = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Inject = No;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ResetHDA = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Properties = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "PciRoot(0x0)/Pci(0x2,0x0)" = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "AAPL,ig-platform-id" = {length = 4, bytes = 0x0300923e};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "framebuffer-patch-enable" = {length = 4, bytes = 0x01000000};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "framebuffer-stolenmem" = {length = 4, bytes = 0x00003001};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> USB = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FixOwnership = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> HighCurrent = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Inject = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> GUI = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Scan = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Entries = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Tool = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Theme = embedded;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Graphics = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Inject = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ATI = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Intel = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> NVidia = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> KernelAndKextPatches = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> AppleIntelCPUPM = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> KernelPm = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> KextsToPatch = (</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Comment = "External icons patch";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Disabled = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Find = {length = 8, bytes = 0x45787465726e616c};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> InfoPlistPatch = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Name = AppleAHCIPort;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Replace = {length = 8, bytes = 0x496e7465726e616c};</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> },</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> );</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> RtVariables = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BooterConfig = 0x28;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> CsrActiveConfig = 0x3E7;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> MLB = C02726902CDH69F1M;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ROM = UseMacAddr0;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> SMBIOS = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BiosReleaseDate = "10/30/2019";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BiosVendor = "Apple Inc.";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BiosVersion = "IM191.88Z.F000.B00.1910301745";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BoardManufacturer = "Apple Inc.";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BoardType = 10;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> BoardVersion = "1.0";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ChassisAssetTag = "iMac-Aluminum";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ChassisManufacturer = "Apple Inc.";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ChassisType = 0x09;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> EfiVersion = "1037.60.50.0.0";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Family = iMac;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FirmwareFeatures = 0xFD8FF576;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FirmwareFeaturesMask = 0xFFDFFF7F;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> LocationInChassis = "Part Component";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Manufacturer = "Apple Inc.";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Mobile = NO;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> PlatformFeature = 0x22;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ProductName = "iMac19,1";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Version = "1.0";</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> SystemParameters = {</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> InjectKexts = Yes;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> InjectSystemID = YES;</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> };</span><br />
<span style="color: #1c1e29; font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<div style="color: #1c1e29;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div style="color: #1c1e29;">
<br /></div>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<br />
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
</div>
<br />
<div style="background: transparent; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-1504526781289431892019-12-28T13:01:00.003-08:002022-07-02T19:14:12.952-07:00Setting up Wireguard on a home linux server<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Wireguard is a peer-to-peer VPN solution with manual IP assignment and pre created keys, so it works well if you want to dial home to your home network, but is not really suited for something large scale that requires dynamic allocation and user management.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h2 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Step 1 - set up the server on Ubuntu</span></h2>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<h4>
<b>Enable IP forwarding</b></h4>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
To have access the outside network through your server once you dial home.</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Run first</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">sysctl -w net.ipv4.ip_forward=1</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Then edit</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">/etc/sysctl.conf</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">and uncomment the next line to enable packet forwarding for IPv4</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">net.ipv4.ip_forward=1</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<h4 style="text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><b>Install Wireguard for Ubuntu</b></span></h4>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new", courier, monospace;">sudo add-apt-repository ppa:wireguard/wireguard</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">apt install wireguard</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<h4 style="text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><b>Generate private and public keys</b></span></h4>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
# generate private key</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">wg genkey > example.key</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"># generate public key</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">wg pubkey < example.key > example.key.pub</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Take note of the content of example.key.pub, you will need it for the client.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<h4 style="text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><b>Enable the Wireguard network interface</b></span></h4>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new", courier, monospace;">sudo systemctl enable wg-quick@wg0</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<h4 style="text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><b>Start the Wireguard interface</b></span></h4>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new", courier, monospace;">wg-quick down wg0; wg-quick up wg0</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Edit <span style="font-family: "courier new" , "courier" , monospace;">/etc/wireguard/wg0.conf</span> to complete the missing sections</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;"><b>[Interface]</b></span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span><b><span style="font-family: courier new, courier, monospace;">A</span><span style="font-family: courier;">ddress =</span></b><span style="font-family: courier;"> 10.10.0.1/24</span></span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: courier;"><b>SaveConfig =</b> true</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;"><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: courier;"><b>PostUp </b>= iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o INTF0 -j MASQUERADE</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: courier;"><b>PostDown </b>= iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o </span><span style="font-family: courier;">INTF0 </span><span style="font-family: courier;">-j MASQUERADE</span></div></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;"><b>ListenPort =</b> 51820</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;"><b>PrivateKey =</b> SHOULDBEHEREALREADY=</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Replace <i>INTF0</i> with your actual network card that faces the internet (e.g., enp3s0).</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The address will be your server's address for wg0. PostUp and PostDown commands enable IP forwarding for the clients.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<h2 style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Step 2 - set up the client</span></h2>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Download an official Wireguard app from </span><a class="_e75a791d-denali-editor-page-rtfLink" href="https://www.wireguard.com/install/" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #4a6ee0; margin-bottom: 0pt; margin-top: 0pt;" target="_blank"><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">https://www.wireguard.com/install/</span></a></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">In the client app, modify the config to complete it to something like this</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">[Interface]</span></strong></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">PrivateKey</strong><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> = SHOULDBEHEREALREADY=</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Address</strong><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> = 10.10.0.2/32</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">DNS</strong><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> = 1.1.1.1</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">[Peer]</span></strong></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">PublicKey</strong><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> = SERVERSPUBLICKEYFROM_EXAMPLE_PUB_KEY=</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">AllowedIPs</strong><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> = 0.0.0.0/0, ::/0</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "courier new" , "courier" , monospace;"><strong style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">Endpoint</strong><span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"> = myregistered.noip.com:51820</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">It's a good idea to register a dynamic DNS address for your endpoint and <a href="https://github.com/adam-ah/noip-updater" target="_blank">auto-update it with a script</a>.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><br /></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The address will be the client's address on the Wireguard's network, and the DNS will be used for that network for name resolution.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The peer is the server, so we need to add the server's public key to the client. AllowedIPs is telling the client what traffic to route through to the server (in this case, all traffic will be routed to the server).</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;">The Endpoint is the server's public IP address or domain name.</span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<h4 style="text-align: left;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><b>Last important step - add the client's config to the server</b></span></h4>
</div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
On the server, run </div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: "courier new" , "courier" , monospace;">sudo wg set wg0 peer CLIENTSPUBLICKEY= allowed-ips 10.10.0.2/32</span></span></div>
<div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
<span style="color: #1c1e29;">This will allow the server to accept the client and route the traffic back to it that belongs to it, once connected.</span><br />
<div style="background: transparent; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">
<span data-preserver-spaces="true" style="background: transparent; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;"><br /></span>
<br />
<h2 style="text-align: left;"><span data-preserver-spaces="true" style="background: transparent; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">Step 3 - on client, route only local / LAN traffic to Wireguard</span></h2>
Calculate the correct <a href="http://www.subnet-calculator.com/" target="_blank">network mask</a> instead of using the 0/0 mask ,and put that into the AllowedIPs.<br />
Remove the DNS entry, otherwise name resolution won't work.</div><div style="background: transparent; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;"><br /></div><div style="background: transparent; color: #1c1e29; margin-bottom: 0pt; margin-top: 0pt;">Something like this:</div><div style="background: transparent; margin-bottom: 0pt; margin-top: 0pt;"><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29;"><br /></span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">[Interface]</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">PrivateKey = efghi12234=</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">Address = 10.10.0.10/32</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;"><br /></span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">[Peer]</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">PublicKey = abcd1234=</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">AllowedIPs = 10.0.0.0/24</span></div><div style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #1c1e29; font-family: courier;">Endpoint = vpn.your.domain.com:51820</span></div><div style="color: #1c1e29;"><br /></div></div>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-18952894407945026572019-07-18T14:50:00.001-07:002019-07-18T15:05:36.530-07:00My LaTeX workflow - from Google Docs to PDF<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://blog.teamleadnet.com/2018/05/generating-apa-6-formatting-and.html">As much as I enjoy writing my academic work in LaTeX</a>, it had a steep learning curve and took a significant time to set up all my tools to work conveniently with it.<br />
<br />
<b>Cloud</b><br />
<br />
One of my requirements was to be able to edit my documents anywhere and potentially collaborate with others on them. While <a href="https://overleaf.com">Overleaf</a> is an excellent tool, it's not a great platform to write long texts in, and the collaboration features are somewhat limited compared to Google Doc.<br />
Google Doc was missing <a href="https://github.com/adam-ah/googledocs_latex_syntaxhighlight">LaTeX syntax highlights, so I created a plugin to do exactly that</a>.<br />
<br />
<b>Tables</b><br />
<br />
Everyone who's using LaTeX agrees that editing tables is as painful as it can get. I like storing my data in Google Sheets, so I created a <a href="https://github.com/adam-ah/google_sheets_to_latex">plugin that converts from Google Sheets to LaTex</a>.<br />
<br />
<b>Reference management</b><br />
<br />
I chose Zotero with <a href="http://retorque.re/zotero-better-bibtex/installation/">Better Bibtex</a> for managing my references, which is a great toolset, but I ran into the problem several times: author names across papers were slightly misspelled, and it caused rendering artefacts when creating the references section in my document. To fix this, I created a <a href="https://github.com/adam-ah/zotero-detect-duplicate-authors">plugin that detects duplicate authors in Zotero</a> and highlights the ones that are potentially the same with slightly different spelling, so I can manually fix them.<br />
<br />
<b>Creating the PDF locally</b><br />
<br />
Finally, I needed to download from Google Docs and generate the final PDF locally - copy-pasting was inconvenient, so I created a tool that can <a href="https://github.com/adam-ah/gdocs_download">download a Google Doc (with all changes accepted) locally</a>, so I can render the final PDF.<br />
<br />
<b>Overall workflow</b><br />
<br />
It might be a bit hard to follow it in text, so here is my full workflow:<br />
<br /></div>
</div>
<svg
viewBox="0.00 0.00 1105.00 676.49" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 672.4939)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-672.4939 1101,-672.4939 1101,4 -4,4"/>
<g id="clust1" class="cluster">
<title>cluster_cloud</title>
<polygon fill="none" stroke="#000000" points="573,-299.4156 573,-660.4939 1089,-660.4939 1089,-299.4156 573,-299.4156"/>
<text text-anchor="middle" x="831" y="-643.8939" font-family="Times,serif" font-size="14.00" fill="#000000">Cloud</text>
</g>
<g id="clust2" class="cluster">
<title>cluster_gd</title>
<polygon fill="none" stroke="#000000" points="809,-307.4156 809,-528.0626 1081,-528.0626 1081,-307.4156 809,-307.4156"/>
<text text-anchor="middle" x="945" y="-511.4626" font-family="Times,serif" font-size="14.00" fill="#000000">Google Doc</text>
</g>
<g id="clust3" class="cluster">
<title>cluster_gs</title>
<polygon fill="none" stroke="#000000" points="581,-439.8469 581,-627.6939 801,-627.6939 801,-439.8469 581,-439.8469"/>
<text text-anchor="middle" x="691" y="-611.0939" font-family="Times,serif" font-size="14.00" fill="#000000">Google Sheet</text>
</g>
<g id="clust4" class="cluster">
<title>cluster_localinstall</title>
<polygon fill="none" stroke="#000000" points="8,-8 8,-428.4313 565,-428.4313 565,-8 8,-8"/>
<text text-anchor="middle" x="286.5" y="-411.8313" font-family="Times,serif" font-size="14.00" fill="#000000">Local installation</text>
</g>
<g id="clust5" class="cluster">
<title>cluster_zotero</title>
<polygon fill="none" stroke="#000000" points="16,-152 16,-395.6313 399,-395.6313 399,-152 16,-152"/>
<text text-anchor="middle" x="207.5" y="-379.0313" font-family="Times,serif" font-size="14.00" fill="#000000">Zotero</text>
</g>
<!-- gd -->
<g id="node1" class="node">
<title>gd</title>
<ellipse fill="none" stroke="#000000" stroke-width="2" cx="879" cy="-333.4156" rx="61.9408" ry="18"/>
<text text-anchor="middle" x="879" y="-329.2156" font-family="Times,serif" font-size="14.00" fill="#000000">Google Docs</text>
</g>
<!-- gdownload -->
<g id="node10" class="node">
<title>gdownload</title>
<g id="a_node10"><a xlink:href="https://github.com/adam-ah/gdocs_download" xlink:title="gdocs-download" target="_blank">
<ellipse fill="none" stroke="#000000" stroke-width="2" cx="482" cy="-250" rx="74.7265" ry="18"/>
<text text-anchor="middle" x="482" y="-245.8" font-family="Times,serif" font-size="14.00" fill="#0000ff">gdocs-download</text>
</a>
</g>
</g>
<!-- gd->gdownload -->
<g id="edge4" class="edge">
<title>gd->gdownload</title>
<path fill="none" stroke="#000000" d="M828.7197,-322.851C757.588,-307.9052 627.4597,-280.5633 548.4388,-263.9598"/>
<polygon fill="#000000" stroke="#000000" points="549.0948,-260.5213 538.5888,-261.8902 547.6554,-267.3717 549.0948,-260.5213"/>
</g>
<!-- gs -->
<g id="node2" class="node">
<title>gs</title>
<ellipse fill="none" stroke="#000000" stroke-width="2" cx="710" cy="-465.8469" rx="63.164" ry="18"/>
<text text-anchor="middle" x="710" y="-461.6469" font-family="Times,serif" font-size="14.00" fill="#000000">Google Sheet</text>
</g>
<!-- gs->gd -->
<g id="edge3" class="edge">
<title>gs->gd</title>
<path fill="none" stroke="#000000" d="M731.7478,-448.805C761.6628,-425.3631 815.6167,-383.0839 849.2497,-356.7285"/>
<polygon fill="#000000" stroke="#000000" points="851.6233,-359.3152 857.3357,-350.3922 847.3056,-353.8053 851.6233,-359.3152"/>
</g>
<!-- lm -->
<g id="node3" class="node">
<title>lm</title>
<ellipse fill="none" stroke="#000000" stroke-width="2" cx="399" cy="-106" rx="42.2932" ry="18"/>
<text text-anchor="middle" x="399" y="-101.8" font-family="Times,serif" font-size="14.00" fill="#000000">latexmk</text>
</g>
<!-- pdf -->
<g id="node6" class="node">
<title>pdf</title>
<polygon fill="none" stroke="#000000" points="437.9431,-52 360.0569,-52 360.0569,-16 437.9431,-16 437.9431,-52"/>
<text text-anchor="middle" x="399" y="-29.8" font-family="Times,serif" font-size="14.00" fill="#000000">.pdf output</text>
</g>
<!-- lm->pdf -->
<g id="edge8" class="edge">
<title>lm->pdf</title>
<path fill="none" stroke="#000000" d="M399,-87.8314C399,-80.131 399,-70.9743 399,-62.4166"/>
<polygon fill="#000000" stroke="#000000" points="402.5001,-62.4132 399,-52.4133 395.5001,-62.4133 402.5001,-62.4132"/>
</g>
<!-- tex -->
<g id="node4" class="node">
<title>tex</title>
<polygon fill="none" stroke="#000000" points="485,-196 431,-196 431,-160 485,-160 485,-196"/>
<text text-anchor="middle" x="458" y="-173.8" font-family="Times,serif" font-size="14.00" fill="#000000">.tex</text>
</g>
<!-- tex->lm -->
<g id="edge6" class="edge">
<title>tex->lm</title>
<path fill="none" stroke="#000000" d="M443.1118,-159.8314C436.0358,-151.1962 427.4592,-140.7299 419.7569,-131.3304"/>
<polygon fill="#000000" stroke="#000000" points="422.3183,-128.9341 413.2728,-123.4177 416.9039,-133.3709 422.3183,-128.9341"/>
</g>
<!-- bib -->
<g id="node5" class="node">
<title>bib</title>
<polygon fill="none" stroke="#000000" points="372,-196 318,-196 318,-160 372,-160 372,-196"/>
<text text-anchor="middle" x="345" y="-173.8" font-family="Times,serif" font-size="14.00" fill="#000000">.bib</text>
</g>
<!-- bib->lm -->
<g id="edge7" class="edge">
<title>bib->lm</title>
<path fill="none" stroke="#000000" d="M358.6265,-159.8314C365.0374,-151.2835 372.7941,-140.9412 379.7883,-131.6156"/>
<polygon fill="#000000" stroke="#000000" points="382.7367,-133.5177 385.9367,-123.4177 377.1367,-129.3177 382.7367,-133.5177"/>
</g>
<!-- zot -->
<g id="node7" class="node">
<title>zot</title>
<g id="a_node7"><a xlink:href="https://www.zotero.org/" xlink:title="Zotero app" target="_blank">
<ellipse fill="none" stroke="#000000" stroke-width="2" cx="326" cy="-250" rx="53.2723" ry="18"/>
<text text-anchor="middle" x="326" y="-245.8" font-family="Times,serif" font-size="14.00" fill="#0000ff">Zotero app</text>
</a>
</g>
</g>
<!-- zot->bib -->
<g id="edge11" class="edge">
<title>zot->bib</title>
<path fill="none" stroke="#000000" d="M330.7945,-231.8314C332.8489,-224.0463 335.296,-214.7729 337.5756,-206.1347"/>
<polygon fill="#000000" stroke="#000000" points="340.9735,-206.9753 340.1409,-196.4133 334.2052,-205.1892 340.9735,-206.9753"/>
</g>
<!-- zbb -->
<g id="node8" class="node">
<title>zbb</title>
<g id="a_node8"><a xlink:href="http://retorque.re/zotero-better-bibtex/installation/" xlink:title="Better bibtex \n (plugin)" target="_blank">
<ellipse fill="none" stroke="#000000" cx="326" cy="-333.4156" rx="65.3959" ry="29.3315"/>
<text text-anchor="middle" x="326" y="-337.6156" font-family="Times,serif" font-size="14.00" fill="#0000ff">Better bibtex </text>
<text text-anchor="middle" x="326" y="-320.8156" font-family="Times,serif" font-size="14.00" fill="#0000ff"> (plugin)</text>
</a>
</g>
</g>
<!-- zbb->zot -->
<g id="edge10" class="edge">
<title>zbb->zot</title>
<path fill="none" stroke="#000000" d="M326,-303.7715C326,-291.9708 326,-278.7604 326,-268.3866"/>
</g>
<!-- zdd -->
<g id="node9" class="node">
<title>zdd</title>
<g id="a_node9"><a xlink:href="https://github.com/adam-ah/zotero-detect-duplicate-authors" xlink:title="detect-duplicate-authors \n (plugin)" target="_blank">
<ellipse fill="none" stroke="#000000" cx="133" cy="-333.4156" rx="109.381" ry="29.3315"/>
<text text-anchor="middle" x="133" y="-337.6156" font-family="Times,serif" font-size="14.00" fill="#0000ff">detect-duplicate-authors </text>
<text text-anchor="middle" x="133" y="-320.8156" font-family="Times,serif" font-size="14.00" fill="#0000ff"> (plugin)</text>
</a>
</g>
</g>
<!-- zdd->zot -->
<g id="edge9" class="edge">
<title>zdd->zot</title>
<path fill="none" stroke="#000000" d="M190.9064,-308.3882C224.3276,-293.9434 265.1432,-276.3026 293.1081,-264.216"/>
</g>
<!-- gdownload->tex -->
<g id="edge5" class="edge">
<title>gdownload->tex</title>
<path fill="none" stroke="#000000" d="M475.9438,-231.8314C473.3488,-224.0463 470.2576,-214.7729 467.3782,-206.1347"/>
<polygon fill="#000000" stroke="#000000" points="470.6205,-204.7933 464.1378,-196.4133 463.9797,-207.0069 470.6205,-204.7933"/>
</g>
<!-- gdhighlight -->
<g id="node11" class="node">
<title>gdhighlight</title>
<g id="a_node11"><a xlink:href="https://github.com/adam-ah/googledocs_latex_syntaxhighlight" xlink:title="gdocs-latex-syntax-highlight \n (plugin)" target="_blank">
<ellipse fill="none" stroke="#000000" cx="945" cy="-465.8469" rx="127.7142" ry="29.3315"/>
<text text-anchor="middle" x="945" y="-470.0469" font-family="Times,serif" font-size="14.00" fill="#0000ff">gdocs-latex-syntax-highlight </text>
<text text-anchor="middle" x="945" y="-453.2469" font-family="Times,serif" font-size="14.00" fill="#0000ff"> (plugin)</text>
</a>
</g>
</g>
<!-- gdhighlight->gd -->
<g id="edge1" class="edge">
<title>gdhighlight->gd</title>
<path fill="none" stroke="#000000" d="M930.3525,-436.4561C917.3644,-410.3951 898.815,-373.1751 887.9289,-351.3319"/>
</g>
<!-- gslatex -->
<g id="node12" class="node">
<title>gslatex</title>
<g id="a_node12"><a xlink:href="https://github.com/adam-ah/google_sheets_to_latex" xlink:title="google-sheets-to-latex \n (plugin)" target="_blank">
<ellipse fill="none" stroke="#000000" cx="691" cy="-565.4782" rx="101.5017" ry="29.3315"/>
<text text-anchor="middle" x="691" y="-569.6782" font-family="Times,serif" font-size="14.00" fill="#0000ff">google-sheets-to-latex </text>
<text text-anchor="middle" x="691" y="-552.8782" font-family="Times,serif" font-size="14.00" fill="#0000ff"> (plugin)</text>
</a>
</g>
</g>
<!-- gslatex->gs -->
<g id="edge2" class="edge">
<title>gslatex->gs</title>
<path fill="none" stroke="#000000" d="M696.6492,-535.8553C699.8652,-518.9913 703.79,-498.4109 706.5491,-483.9426"/>
</g>
</g>
</svg>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-7909142166334767602018-09-23T02:13:00.003-07:002024-02-17T23:17:48.838-08:00Machine learning: Prediction suicidal behaviour based on drug abuse and mental health<div dir="ltr" style="text-align: left;" trbidi="on">
There is some well-known correlation between certain mental disorders and suicidal ideation or suicidal behaviour.<br />
<br />
I was interested in whether a machine learning model could be trained to identify suicidal behaviour based on mental health, biological health, and drug abuse questions.<br />
<br />
The dataset I was working with came from the public <a href="https://www.icpsr.umich.edu/icpsrweb/NAHDAP/studies/27963/version/1" rel="nofollow" target="_blank">Criminal Justice Drug Abuse Treatment Studies (CJ-DATS): The Criminal Justice Co-Occurring Disorder Screening Instrument (CJ-CODSI)</a> and had new 353 new admissions to a prison-based substance abuse treatment program (137 Whites, 96 African Americans, and 120 Latinos).<br />
The questionnaire overall was very detailed with 789 attributes, either direct data input or compound fields, based on other fields.<br />
<br />
The variable I wanted to forecast was called SATTLF (Suicide Attempts Lifetime) and to make sure that no suicide or suicide-related questions were left in the source data, I removed 164 fields (including SATTLF), which directly or indirectly referred to suicide.<br />
<br />
I built two models, a simple decision tree and a random forest <a href="http://scikit-learn.org/" rel="nofollow" target="_blank">scikit-learn</a> Python module.<br />
<br />
<h2 style="text-align: left;">
Building the models</h2>
<br />
The models were reasonably easy to build, although the optimal number of estimators had to be fine-tuned for the random forest model. Interestingly, the random forest did not outperform the simple decision tree as much as I thought it would have.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import numpy as np</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import pandas as pd</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from sklearn.model_selection import train_test_split</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from sklearn.ensemble import RandomForestClassifier</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from sklearn.tree import DecisionTreeClassifier</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">data = pd.read_csv('27963-0002-Data.tsv', sep='\t')</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">y = data['SATTLF'] == 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">feature_names = [i for i in data.columns if data[i].dtype in [np.int64]]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">suicide_features = ['SATTLF', 'mhsf28a', [...] 'oaccodss']</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">for sf in suicide_features:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> feature_names.remove(sf)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">X = data[feature_names]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1, test_size=0.2)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">my_model = RandomForestClassifier(random_state=0, warm_start=True, oob_score=True, n_estimators=3, max_features="auto").fit(train_X, train_y)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">my_dec_tree_model = DecisionTreeClassifier(random_state=0, max_depth=10, min_samples_split=5, min_samples_leaf=3).fit(train_X, train_y)</span><br />
<br />
<h2 style="text-align: left;">
Results</h2>
<br />
Overall, the models were reasonably accurate, compared to the difficulty of the problems. The decision tree had an F1 score of 0.6 (<i>precision: 0.562, recall: 0.643</i>), while the random forest had an F1 score of 0.643 (<i>precision: 0.643, recall: 0.643</i>). In other words, both models correctly identified 9 people out of 14 (<i>64.3%</i>) who acted on their suicidal thoughts (total number of people in the validation dataset: 71). It is important to highlight, that the model did not just identify suicidal thoughts, but actual behaviour, as in, acting on these thoughts, which is more difficult to predict.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5dbdNlgxHdHhHbOPBCm1vvKFPQY_QY7Wns0E92_LiRD7i19QXZ5LPiF2kykLPp2ClAdYL-1yA-9aC6PX0doIqLnxCmtHGb7Jjf4yWy70gRvEGbFifPN9nrb0Iy8rr3NHbspu2T0O8yMk/s1600/decisiontree.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="625" data-original-width="1200" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5dbdNlgxHdHhHbOPBCm1vvKFPQY_QY7Wns0E92_LiRD7i19QXZ5LPiF2kykLPp2ClAdYL-1yA-9aC6PX0doIqLnxCmtHGb7Jjf4yWy70gRvEGbFifPN9nrb0Iy8rr3NHbspu2T0O8yMk/s320/decisiontree.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Decision tree from my_dec_tree_model</td></tr>
</tbody></table>
<br />
<br />
Some attributes in the random forest that correlated to the final score (note: none of these in themselves are enough for a prediction, their interaction matters)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj938UC471b-IL885uPzb-qY6ydRl2IQQOCE4Wtsp0ZEHtc9xIHTvaRdqOEJosR22USRLeEr__dm6xO1BOoQhlAyPPgE7PUnc2tJ9xBGuzUDY7ty0nPS7GhyphenhyphendPS7s1wouz8X3mGxiCMTD0/s1600/SHAPsummary.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="582" data-original-width="543" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj938UC471b-IL885uPzb-qY6ydRl2IQQOCE4Wtsp0ZEHtc9xIHTvaRdqOEJosR22USRLeEr__dm6xO1BOoQhlAyPPgE7PUnc2tJ9xBGuzUDY7ty0nPS7GhyphenhyphendPS7s1wouz8X3mGxiCMTD0/s400/SHAPsummary.png" width="371" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">SHAP summary for my_model (random forest)</td></tr>
</tbody></table>
<br />
<br />
<b>psychhosp</b> - Psychiatric hospitalisations (0 / 1). A low value (0) was a moderator, while high value (1) seemed to significantly boost the suicide attempt score.<br />
<br />
<b>mhsfto13</b> - Score of 13 or higher on MHSF (18 different psychopathology questions). A low value (0) was a moderator, while high value (1) seemed to boost the suicide attempt score.<br />
<br />
<b>tcuds</b> - Drug use questionnaire, score between 0-9. Higher scores seemed to boost the suicide attempt score.<br />
<br />
<b>AGE1HER</b> - Age 1st Heroin Use, low score (missing values) is a protective factor.<br />
<br />
<b>pdborder</b> - Personality Disorders- Borderline, score between 0-9. Low and middle scores seemed to be neutral, higher scores seemed to significantly contribute to the overall score.<br />
<br />
A sample SHAP plot for a subject in the dataset:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuMmIRA2fn3LE8woNXy7b3wSDOmWCdYoPXZKU5vx5tL9ljs1fjW_S-qGGMke_apMc7n5hG4E7UXDI1MVW-HMV5fWm4B1wysrBVI9zVpOgwledf36l4x_PZGiN1BZQgr63hZ-_NJ5MaY0c/s1600/SHAPplot.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="152" data-original-width="998" height="60" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuMmIRA2fn3LE8woNXy7b3wSDOmWCdYoPXZKU5vx5tL9ljs1fjW_S-qGGMke_apMc7n5hG4E7UXDI1MVW-HMV5fWm4B1wysrBVI9zVpOgwledf36l4x_PZGiN1BZQgr63hZ-_NJ5MaY0c/s400/SHAPplot.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Single SHAP plot for subject #47</td></tr>
</tbody></table>
<br />
<br />
As apparent, physiatric hospitalisation, heroin use at age of 18, and 5 out of 9 borderline behaviours significantly contributed to the overall score. On the other hand, not using crack in the last 6 months, finishing 12 grade, and scoring only 3 on the TCUDS drug questionnaire were protective factors, yielding an overall score of 0.33.<br />
<br />
<h2 style="text-align: left;">
Dependence plots</h2>
<br />
There were two interesting attributes that seemed to have a cutoff value: the highest grade finished and borderline behaviour score. It appears that finishing 11th grade or higher, or having a score of 4 or lower on the borderline behaviour scale is a protective factor in terms of suicide behaviour.<br />
<br />
What is salient, and difficult to explain, is that according to the <a href="https://www.psychiatry.org/psychiatrists/practice/dsm" rel="nofollow" target="_blank">Diagnostic and Statistical Manual of Mental Disorders (DSM–5)</a>, a person meets the diagnostic criteria for borderline personality disorder if their behaviour matches 5 or more out of the 9 behavioural patterns that are typical with this disorder. It is interesting, that t<b>he model built on the data in our case had that significant increase exactly where the DSM-5 draws the line, even thought there is nothing specific about that value.</b> One explanation could be, that clinicians intentionally rounded up values to 5 if they felt that the patient met the overall diagnosis; however, there are ample of inmates with value of 4, so this might not hold. This finding warrants further research.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHihSfrJGJ293k4OuxM1O4q7lJG579jUd0IRkVrQeB6k761f5n41WZDXk-IFdlA5b5qvksnJELIJfIHoyiIFGVp3Qf-olnXkg11x9VFhIMwLsQvkwFa3Wzxi9bqXuQ9vXmwT-mlDF4oHI/s1600/SHAPdependence.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="669" data-original-width="502" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHihSfrJGJ293k4OuxM1O4q7lJG579jUd0IRkVrQeB6k761f5n41WZDXk-IFdlA5b5qvksnJELIJfIHoyiIFGVp3Qf-olnXkg11x9VFhIMwLsQvkwFa3Wzxi9bqXuQ9vXmwT-mlDF4oHI/s1600/SHAPdependence.png" /></a></div>
<br />
<br />
<h2 style="text-align: left;">
Code</h2>
<br />
The full source code is available as a <a href="https://github.com/adam-ah/predict-sc" rel="nofollow" target="_blank">Python notebook on Github</a>.<br />
<br />
<h2 style="text-align: left;">
<u>
Reach out</u></h2>
<div>
<span style="font-size: large;"><br /></span></div>
<b><span style="font-size: large;">If you have suicidal thoughts, give a call to a local support centre in your country. <a href="https://personalpsychology.com.au" target="_blank">Personal Psychology psychologists in North Sydney</a>, also offers therapy, if you are local.</span></b><br />
<b><span style="font-size: large;">You can <a href="https://www.google.com/search?q=suicidal+thoughts" rel="nofollow" target="_blank">find phone numbers here</a> or <a href="http://ibpf.org/resource/list-international-suicide-hotlines" rel="nofollow" target="_blank">here</a>.</span></b><br />
<br /></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-27584821412482818262018-07-23T19:42:00.000-07:002018-09-23T02:01:24.099-07:00Zero knowledge AI for Tic-tac-toe and Gomoku (Five in a row) games<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Starting from zero</b><br />
<br />
When building AI for games, zero knowledge simply means that when the system starts learning the game it only understands what it means to move, win, or lose. In certain situations, it is certainly easier and faster to simply instruct the machine how to play instead of building a system that can infer knowledge from a gameplay: for simple or very fixed problems, like Tic-tac-toe, an algorithmic approach can be an excellent solution.<br />
<br />
However, as the game complexity increases it is not always easy to program the system how to play - sometimes advanced players can employ different strategies and beat the computer all the time. In these situations, a self-learning system might be a better approach.<br />
<br />
<b>Artificial neural networks</b><br />
<br />
Neural networks are great to recognise complex patterns and build predictions based on these patterns, although it's worth keeping in mind that they aren't the best solution to every problem: as the "knowledge" is stored in matrix weights, usually most if not all intuition is lost as of why a certain type of prediction was generated based on some input data.<br />
<br />
Generally speaking, ANNs are great choices in situations where the rules can be very complex and some prediction error is acceptable. On the other hand, when the rules are reasonably simple (eg, can be phrased in simple human language) other approaches, like a manually created set of rules, would be potentially better for solving the problem.<br />
<br />
<b>Generate training data</b><br />
<br />
To teach ANNs we need as much training data as we can get. Ideally, we would have access to some expert players' previous games and start from there but that's not always feasible - maybe we are inventing a new game and want to see what strategies would be good, or simply don't have access to expert gameplays.<br />
<br />
Also, the training data should be labelled so the system can learn what is a "better" or "worse" move - simply showing gameplays without proper labels will make the data impossible to learn.<br />
Imagine the following gameplay:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw5rXR1yiseOm98yhJEq9VC7acLnK7aHp4I_Rga1eqIhrSmYXogbSpxDIfLWhjR_P-v_oVw_sDCohx5lAAgHfMjaA03DjYdyUupgGcY6zSJEgLuK4Td6ngVcgGlH4uQg-hgO1yRZlgLxQ/s1600/dump_tictactoe.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="764" data-original-width="964" height="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw5rXR1yiseOm98yhJEq9VC7acLnK7aHp4I_Rga1eqIhrSmYXogbSpxDIfLWhjR_P-v_oVw_sDCohx5lAAgHfMjaA03DjYdyUupgGcY6zSJEgLuK4Td6ngVcgGlH4uQg-hgO1yRZlgLxQ/s320/dump_tictactoe.png" width="320" /></a></div>
<br />
It is very obvious to a human player that X could have won the game so depending on how we label the data we could either conclude that placing the X on the middle position will make X lose the game, or we could conclude that the 3rd step of X was a terrible idea. It all depends how we label the data across many gameplays.<br />
<br />
Obviously, the naive approach to label the steps solely based on who won isn't right. What we could do instead to label each move based on how good idea it has been during previous games, so eventually the labels will be correct percentages of winning, based on that setup.<br />
<br />
<b>Labelling based on the Monte Carlo Tree</b><br />
<br />
The <a href="https://en.wikipedia.org/wiki/Monte_Carlo_tree_search" rel="nofollow" target="_blank">Monte Carlo Tree Search</a> was invented to guide AI of computer games based on previous successes and failures of the same move by adding some random to it to explore new ideas, in case those might be better than the current ones.<br />
<br />
For our gameplay we aren't using the MCTS but we use it for generating the training data - after each game, we update the boards in our training dataset for successes and failures counts and retrain our model based on the updated values.<br />
<br />
Even if based on the first game above the middle X position seems to be a losing position, eventually, over many-many games, it will have a higher "win" ratio than a "draw" or a "lose" ratio and our model will learn that it is the best next move from an empty board. This logic applies to all moves so after enough training data the correctly trained model would learn that in similar situations what was the best move.<br />
<br />
And the "similar situations" is the key here - while the MCTS cannot answer similarity questions, our artificial neural network can. It will learn what patterns generally lead to winning and losing and the predicted weights will reflect that.<br />
<br />
<b>Randomisation</b><br />
<br />
When our model starts learning it knows nothing about the game - all predicted moves are probably incorrect (although not random, the network will have some initial weights and it will consistently predict the poor moves, not randomly).<br />
<br />
The first question is, should we generate totally random games, label those moves and train the model first on that, or should we play the model against a somewhat random itself right away?<br />
<br />
Also, when the model is playing against itself, how random should be the opponent player?<br />
To answer the first question, it seems like that the more complex the game, the worse idea it is as the totally random moves will significantly distort the weights - just think about our previous example. For an 8x8 board where 5 in a row required to win we have 3^64 potential board setups, which is 3.43e30. The random games will be so diverse that it would be very difficult to learn from those.<br />
<br />
The second question is a bit trickier - one approach to take is to randomise based on how bad the situation is. If the weights predict a guaranteed winner move then don't randomise too much, let's say only 1 in 20 moves (5%). If the prediction is a draw in that situation, randomise more (25%) but if it's a guaranteed losing setup we may as well something new (50%).<br />
<br />
<b>Rotating, flipping, and whose turn it is</b><br />
<br />
The other tricky question is how to store the tables? Looking at the following setup, guess who will win:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuYQZlTecYzrmg-Pu2e_BgB4jn_9M-lvkFHIM9Wj1Ulr1MukrnRQHYGiMAeQzlusYrZWXWHpDJe9Iu8cdc4nriv9cWbQRTJZNGrCVExfAUWzlTQvNdUYbaIm2zN0Rd_uVCbxzwhwHM3Iw/s1600/who_wins.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="219" data-original-width="236" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuYQZlTecYzrmg-Pu2e_BgB4jn_9M-lvkFHIM9Wj1Ulr1MukrnRQHYGiMAeQzlusYrZWXWHpDJe9Iu8cdc4nriv9cWbQRTJZNGrCVExfAUWzlTQvNdUYbaIm2zN0Rd_uVCbxzwhwHM3Iw/s1600/who_wins.png" /></a></div>
<br />
<br />
It depends on whose turn it is! We need to make sure that we store each board in a way that is consistent for learning, so we store it as if it was X's move. If it was O's move, we swap the boards and make sure that it looked like it was X's move. Without doing this the system would need to learn inconsistent values: for the setup above sometimes it's a winning, sometimes it's a losing setup.<br />
<br />
Also, because we know that the game doesn't have a direction (unlike chess), we can flip horizontally and vertically, and rotate the boards 90, 180 and 270 degrees after each game and store the same weights with it, increasing the training data without playing more games.<br />
<br />
<b>Neural Network Architecture</b><br />
<br />
As the game is about local features (N in a row, not anywhere on the board) it seems logical to use convnets. In my case I used the following architecture:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Conv2D > Conv2D > MaxPooling > Conv2D > Conv2D > Flatten > Dense > Dense (1)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
The size (or filter count) for each layer really depends on the problem at hand, but for the board size of 6 and win length of 5 the<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">128 > 128 > 256 > 256 > 128 > 1</span><br />
<br />
setup seemed sufficient.<br />
<br />
One drawback of this approach is that it only evaluates the current board with a value between -1 and 1, but does not predict potential next moves. To get the next move we need to iterate through the board, place a piece on an empty spot, evaluate that setup, and remove the piece. If this is a better situation than the previous, we recommend this move.<br />
<br />
However, for larger boards, this is quite slow which makes the training slow. For the sizes mentioned above we already have 1,068,289 tunable parameters, so predicting a single value for an empty cell is quite computationally expensive, and for each move we need to do it as many times as there are empty cells, making it an <span style="font-family: "courier new" , "courier" , monospace;">O(N^2)</span> approach.<br />
<br />
This can be considered as the "value net" and the <a href="http://nature.com/articles/doi:10.1038/nature24270" rel="nofollow" target="_blank">AlphaGo Zero</a> used a policy net as well, which predicted which moves are worth evaluating at all, dramatically reducing the cost of finding the next best move.<br />
<br />
<b>Does it work?</b><br />
<br />
The model isn't perfect but reasonably good. Here is model (O) beating me (X) in an actual gameplay. It learned very well how to block me from creating a straight line of 5 pieces. In other words, it adapted to the board size very well, as these approaches wouldn't work on a larger board.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbCCeXfO_qtsjxsYh351R8RvuoXT959kqTnOVX_Fr06dRWvMeAQ3jAS6k_JKwGV2jPSL4cPakPF1DOvv0X44mqRyvFuh9QdvFYVVdqXCn6mXOuHhkkX6RtOjE-aDdfeUn0p-_dowFk-MY/s1600/actual_game.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="477" data-original-width="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbCCeXfO_qtsjxsYh351R8RvuoXT959kqTnOVX_Fr06dRWvMeAQ3jAS6k_JKwGV2jPSL4cPakPF1DOvv0X44mqRyvFuh9QdvFYVVdqXCn6mXOuHhkkX6RtOjE-aDdfeUn0p-_dowFk-MY/s1600/actual_game.png" /></a></div>
<br />
<br />
<b>The code</b><br />
<br />
The full source <a href="https://github.com/adam-ho/ai-gomoku" target="_blank">code is available on GitHub</a></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-81116970964533186102018-05-17T01:01:00.001-07:002018-12-30T22:46:18.014-08:00Generating APA 6 formatting and references with LaTeX and BibLaTeX<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
What is LaTeX</h2>
It's a toolset that is able to transform the documents written in TeX language into something beautifully rendered, like a PDF file, requiring almost no effort for formatting. It was invented to counteract the problem of different machines displaying files differently, especially things that are hard to format, like equations, tables, and so on.<br />
<br />
<br />
The steps typically look like this:<br />
<span style="font: 0.5em;"> </span>
<br />
<pre><span style="font-size: x-small;">+----------------+ +------------------+
| Reference Mgr. +------> Biblatex .bib +--------+
+----------------+ +------------------+ +v-------------+ +-----------+
| latexmk +----> .pdf |
+----------------+ +------------------+ +^-------------+ +-----------+
| Text editor +------> LaTeX .tex +--------+
+----------------+ +------------------+</span>
</pre>
</div>
<br />
<h2 style="text-align: left;">
Automate referencing and formatting with LaTeX for psychology papers</h2>
Psychology papers are mostly text heavy with some formulas and tables, and generally with a lot of referencing and citations with special formattings, like APA 6 essay style, or two column formatting. Making sure that the document format and the references are correct is time consuming and error-prone but LaTeX can totally automate this process:<br />
<br />
- Format the document according to the full APA 6 standard<br />
- Automate referencing and bibliography management<br />
- Highlight unused references<br />
- Correctly reference all tables and figures<br />
<br />
The advantage of using automation to generate the correct final APA 6 format is that we can focus on writing the content itself without worrying about accidentally missing a formatting requirement or a reference from bibliography, or correctly pointing to a figure from the text, or just an italics when inserting a mathematical formula. Also, as the source is a simple text file, it's much denser and easier to read than the final output, which according to APA 6 is required to use double spacing, making it hard to overview.<br />
<br />
<h2 style="text-align: left;">
Workflow for reference management</h2>
First, as usual, we collect the references for the paper in either <a href="https://www.zotero.org/" rel="nofollow" target="_blank">Zotero</a> or <a href="https://www.mendeley.com/" rel="nofollow" target="_blank">Mendeley</a>. I personally found a couple of papers that Mendeley had incorrectly imported and the author names were distorted, so I've settled for Zotero, which happens to be open source, unlike Mendeley. Once we have the references, we can export them into BibLaTeX format. I would also highly recommend installing <a href="https://retorque.re/zotero-better-bibtex/installation/" rel="nofollow" target="_blank">Better BibTeX for Zotero</a>, as the BibLaTeX export is not buggy, it allows customising the citation keys, and even copying them from Zotero with a shortcut (Shift+Command+C).<br />
<br />
Editing the LaTeX (.tex) file can be done in any editor, even in Word or LibreOffice as it's a simple text file with some markers but without any kind of formatting. I have attached a sample APA 6 template at the end of this post that will generate a fully formatted APA 6 essay with very little effort from the author.<br />
<br />
To generate the output PDF file, either use an online TeX editor or run the LaTeX toolkit locally (see both later):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">latexmk -pdf -xelatex</span><br />
<br />
<h2 style="text-align: left;">
Getting LaTeX, generating the APA 6 format</h2>
The only difficult part of using LaTeX is actually setting up the environment. There are great online collaborative LaTeX tools that remove this problem altogether, like <a href="https://www.overleaf.com/" target="_blank">Overleaf</a>. It's good to start with those to have a feel for editing TeX files before installing the environment ourselves.<br />
<br />
When we are familiar with TeX, it's good to install it locally too: visit the <a href="https://www.latex-project.org/get/" rel="nofollow" target="_blank">LaTeX project</a> and find the package for the system.<br />
<br />
For Mac manual installation, I decided to install it using <a href="https://www.macports.org/install.php" rel="nofollow" target="_blank">MacPorts</a> instead of the prepackaged <a href="http://www.tug.org/mactex/" rel="nofollow" target="_blank">MacTex</a> installer, so these are the components that I needed:<br />
<br />
<pre>sudo port install texlive</pre>
<pre>sudo port install biblatex-biber
sudo port install texlive-bibtex-extra
sudo port install texlive-publishers
sudo port install texlive-fonts-extra
sudo port install texlive-fonts-recommended
</pre>
<pre></pre>
<br />
Unfortunately, this approach is very time consuming, so retrospectively it would have been much easier to use <a href="http://www.tug.org/mactex/" rel="nofollow" target="_blank">MacTex</a>...<br />
<br />
<h2 style="text-align: left;">
Sample APA 6 TeX file and the output</h2>
The following sample will generate an APA 6 essay in PDF with two "References" section. The second list is the "error list", highlighting the unused references in the BibLaTeX .bib resource - it's not a bug, it's a feature. Try to run it locally or just upload it to Overleaf.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">main.tex</span><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green; font-weight: bold;">\documentclass</span><span style="color: #7d9029;">[a4paper,man,floatsintext]</span><span style="color: green;">{</span>apa6<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\usepackage</span><span style="color: #7d9029;">[american]</span><span style="color: green;">{</span>babel<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\usepackage</span><span style="color: green;">{</span>csquotes<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\usepackage</span><span style="color: #7d9029;">[style=apa,sortcites=true,sorting=nyt,backend=biber]</span><span style="color: green;">{</span>biblatex<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\usepackage</span><span style="color: #7d9029;">[colorlinks,citecolor=blue,urlcolor=blue]</span><span style="color: green;">{</span>hyperref<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\DeclareLanguageMapping</span><span style="color: green;">{</span>american<span style="color: green;">}{</span>american-apa<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\usepackage</span><span style="color: #7d9029;">[utf8]</span><span style="color: green;">{</span>inputenc<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\usepackage</span><span style="color: #7d9029;"></span><span style="color: green;">{</span>fontspec<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\defaultfontfeatures</span><span style="color: #7d9029;"></span><span style="color: green;">{</span>Mapping=tex-text, Scale=MatchLowercase<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\setmainfont</span><span style="color: #7d9029;"></span><span style="color: green;">{</span>Times New Roman<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\DeclareBibliographyCategory</span><span style="color: green;">{</span>cited<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\AtEveryCitekey</span><span style="color: green;">{</span><span style="color: green; font-weight: bold;">\addtocategory</span><span style="color: green;">{</span>cited<span style="color: green;">}{</span><span style="color: green; font-weight: bold;">\thefield</span><span style="color: green;">{</span>entrykey<span style="color: green;">}}}</span>
<span style="color: green; font-weight: bold;">\nocite</span><span style="color: green;">{</span>*<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\addbibresource</span><span style="color: green;">{</span>main.bib<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\title</span><span style="color: green;">{</span>The effect of Social networks on us<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\shorttitle</span><span style="color: green;">{</span>Social networks<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\author</span><span style="color: green;">{</span>Adam Horvath<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\affiliation</span><span style="color: green;">{</span>Your University <span style="color: green; font-weight: bold;">\\</span> studentid12345<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\abstract</span><span style="color: green;">{</span>Your abstract here.<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\begin</span><span style="color: green;">{</span>document<span style="color: green;">}</span>
<span style="color: green; font-weight: bold;">\maketitle</span>
Your research paper goes here. You can inline cite like <span style="color: green; font-weight: bold;">\textcite</span><span style="color: green;">{</span>ryan<span style="color: green;">_</span>who<span style="color: green;">_</span>2011<span style="color: green;">}</span>. But of course, the usual citation works too <span style="color: green; font-weight: bold;">\parencite</span><span style="color: green;">{</span>ryan<span style="color: green;">_</span>who<span style="color: green;">_</span>2011<span style="color: green;">}</span>. Pan es union nostre, con medical linguage historiettas un. Altere auxiliar giuseppe web ha. Del linguas introduction es. Peano publicava linguistic se pro. Su comprende simplificate qui. Sia ma vide germano, tempore specimen per al.
This is a new paragraph. Pan es union nostre, con medical linguage historiettas un. Altere auxiliar giuseppe web ha. Del linguas introduction es. Peano publicava linguistic se pro. Su comprende simplificate qui. Sia ma vide germano, tempore specimen per al.
<span style="color: green; font-weight: bold;">\printbibliography</span><span style="color: #7d9029;">[category=cited]</span>
<span style="color: green; font-weight: bold;">\printbibliography</span><span style="color: #7d9029;">[notcategory=cited]</span>
<span style="color: green; font-weight: bold;">\end</span><span style="color: green;">{</span>document<span style="color: green;">}</span>
</pre>
</div>
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">main.bib <!-- HTML generated using hilite.me --></span><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: "courier new" , "courier" , monospace;">@article<span style="color: green;">{</span>ryan<span style="color: green;">_</span>who<span style="color: green;">_</span>2011,
title = <span style="color: green;">{</span>Who uses Facebook? An investigation into the relationship between the Big Five, shyness, narcissism, loneliness, and Facebook usage<span style="color: green;">}</span>,
volume = <span style="color: green;">{</span>27<span style="color: green;">}</span>,
issn = <span style="color: green;">{</span>07475632<span style="color: green;">}</span>,
url = <span style="color: green;">{</span>http://linkinghub.elsevier.com/retrieve/pii/S0747563211000379<span style="color: green;">}</span>,
doi = <span style="color: green;">{</span>10.1016/j.chb.2011.02.004<span style="color: green;">}</span>,
shorttitle = <span style="color: green;">{</span>Who uses Facebook?<span style="color: green;">}</span>,
pages = <span style="color: green;">{</span>1658--1664<span style="color: green;">}</span>,
number = <span style="color: green;">{</span>5<span style="color: green;">}</span>,
journaltitle = <span style="color: green;">{</span>Computers in Human Behavior<span style="color: green;">}</span>,
author = <span style="color: green;">{</span>Ryan, Tracii and Xenos, Sophia<span style="color: green;">}</span>,
urldate = <span style="color: green;">{</span>2018-05-12<span style="color: green;">}</span>,
date = <span style="color: green;">{</span>2011-09<span style="color: green;">}</span>,
langid = <span style="color: green;">{</span>english<span style="color: green;">}</span>,
<span style="color: green;">}</span>
@article<span style="color: green;">{</span>indian<span style="color: green;">_</span>when<span style="color: green;">_</span>2014,
title = <span style="color: green;">{</span>When Facebook is easier than face-to-face: Social support derived from Facebook in socially anxious individuals<span style="color: green;">}</span>,
volume = <span style="color: green;">{</span>59<span style="color: green;">}</span>,
issn = <span style="color: green;">{</span>01918869<span style="color: green;">}</span>,
url = <span style="color: green;">{</span>http://linkinghub.elsevier.com/retrieve/pii/S0191886913013743<span style="color: green;">}</span>,
doi = <span style="color: green;">{</span>10.1016/j.paid.2013.11.016<span style="color: green;">}</span>,
shorttitle = <span style="color: green;">{</span>When Facebook is easier than face-to-face<span style="color: green;">}</span>,
pages = <span style="color: green;">{</span>102--106<span style="color: green;">}</span>,
journaltitle = <span style="color: green;">{</span>Personality and Individual Differences<span style="color: green;">}</span>,
author = <span style="color: green;">{</span>Indian, Michaelle and Grieve, Rachel<span style="color: green;">}</span>,
urldate = <span style="color: green;">{</span>2018-05-12<span style="color: green;">}</span>,
date = <span style="color: green;">{</span>2014-03<span style="color: green;">}</span>,
langid = <span style="color: green;">{</span>english<span style="color: green;">}</span>,
<span style="color: green;">}</span>
</span></pre>
</div>
<br />
The output PDF (as images). Notice the last page where the missing references are listed (indian_when_2014 was not cited in the text):<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQvYVsOgN0T2TLqfdv8QhuAVTxDkEaqw32KQf1umjBE_Zcuev2WMDcBDQoGYKCS2NIi3xNEtxPpms12gB6PR3fs9oeTfEaA_XP1HZpdqvcXMhAXL_Bgay_7W3vkpzCQWtlOtXSJF8Xi8E/s1600/main-1.png" imageanchor="1"><img border="0" data-original-height="1600" data-original-width="1132" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQvYVsOgN0T2TLqfdv8QhuAVTxDkEaqw32KQf1umjBE_Zcuev2WMDcBDQoGYKCS2NIi3xNEtxPpms12gB6PR3fs9oeTfEaA_XP1HZpdqvcXMhAXL_Bgay_7W3vkpzCQWtlOtXSJF8Xi8E/s200/main-1.png" width="141" /></a> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgK4uQt5dbd8VGrRM_RVU_mHmKtQUFLQbZAeJoay7u_nFlJ5VucTVhga6sH2GEJ-YJsbYP8ZLe1EuAfGwKpM3cLDt8909GovzkFShku5rRmNVTM77uI2zMAtxbb8iUUWNhIfYwJ6q_lcnk/s1600/main-2.png" imageanchor="1"><img border="0" data-original-height="1600" data-original-width="1132" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgK4uQt5dbd8VGrRM_RVU_mHmKtQUFLQbZAeJoay7u_nFlJ5VucTVhga6sH2GEJ-YJsbYP8ZLe1EuAfGwKpM3cLDt8909GovzkFShku5rRmNVTM77uI2zMAtxbb8iUUWNhIfYwJ6q_lcnk/s200/main-2.png" width="141" /></a> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-IHcXkIRUkT0LcMj9qLTkIhOYKzfIl0Y5FiAffth2E-lqVlzPe5iLPrPUlIsYxjARnWDP0B5996D0KWeRwtCQvKFHVsWULZzzLbrrCJnq32O8mUIlknrN2mgZsCSCGldk8sejGxl60YM/s1600/main-3.png" imageanchor="1"><img border="0" data-original-height="1600" data-original-width="1132" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-IHcXkIRUkT0LcMj9qLTkIhOYKzfIl0Y5FiAffth2E-lqVlzPe5iLPrPUlIsYxjARnWDP0B5996D0KWeRwtCQvKFHVsWULZzzLbrrCJnq32O8mUIlknrN2mgZsCSCGldk8sejGxl60YM/s200/main-3.png" width="141" /></a> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAsSJFuiDnBCrgc_cLcU1_1zg0x4pnjkuLz4JBEfIJ9dwpcv5Mefv0el2vc7dPjV4zlOVcOOd555Bbg37pFejgHMN2yeyoF_uDBo5WofaCYI4PERwde18EkCmdU1EMDJZ5nyHcVcvNt5k/s1600/main-4.png" imageanchor="1"><img border="0" data-original-height="1600" data-original-width="1132" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAsSJFuiDnBCrgc_cLcU1_1zg0x4pnjkuLz4JBEfIJ9dwpcv5Mefv0el2vc7dPjV4zlOVcOOd555Bbg37pFejgHMN2yeyoF_uDBo5WofaCYI4PERwde18EkCmdU1EMDJZ5nyHcVcvNt5k/s200/main-4.png" width="141" /></a> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJiAcH5c2pcn6HIZcsSYVJmpVjH7pVYaPgxXGRXKw1GB3X5ieOysecUFgMuOLWpUG475o-vOI9_7koIe8l5-oJDeU3uE07NDYEIbHgoAHNETujdPf9rLxz83poWYAw_Iaq2_rB2jmSNuI/s1600/main-5.png" imageanchor="1"><img border="0" data-original-height="1600" data-original-width="1132" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJiAcH5c2pcn6HIZcsSYVJmpVjH7pVYaPgxXGRXKw1GB3X5ieOysecUFgMuOLWpUG475o-vOI9_7koIe8l5-oJDeU3uE07NDYEIbHgoAHNETujdPf9rLxz83poWYAw_Iaq2_rB2jmSNuI/s200/main-5.png" width="141" /></a>
</div>
<br /></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-11921436506666863452017-08-16T17:26:00.003-07:002017-08-18T18:12:24.940-07:00Building statistical machine translation based on sample bilingual text corpus<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPktV7-zNx00IrBAC5t6vHCe-lA9YCyzQiM44y-UH7A5_CUVgrnjVlASwCO5dSd5tAKOKW_vaPrOmsgTzVLXra7eufMw1LDdI3rww085l_bDFncyJhCtux4lG9rZKHiCUQrk21pfedamg/s1600/g_small.png" imageanchor="1"><img border="0" height="115" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPktV7-zNx00IrBAC5t6vHCe-lA9YCyzQiM44y-UH7A5_CUVgrnjVlASwCO5dSd5tAKOKW_vaPrOmsgTzVLXra7eufMw1LDdI3rww085l_bDFncyJhCtux4lG9rZKHiCUQrk21pfedamg/s200/g_small.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;" width="200" /></a>Training a weighted graph between the source and destination words and phrases in a sample bilingual text corpus yields an acceptable statistical machine translation dictionary.
<br />
<br />
Following up on <a href="http://blog.teamleadnet.com/2017/08/detecting-language-features-and-automatic-translation.html" target="_blank">my previous post about language features</a>, I felt that the simple sentence to sentence translation yielded too few translations, even though they were correct. I wanted to build something that creates a much bigger dictionary, even if it's noisier: false positives are still better than no translation at all.
<br />
<br />
<h2 style="text-align: left;">
Larger graph</h2>
Using the same sample, <a href="https://www.gutenberg.org/files/5201/5201-h/5201-h.htm" rel="nofollow" target="_blank">Don Juan from the Gutenberg project</a> and mapping the Spanish source to the English translation I used a different approach this time: map each word and phrase in the source sentence to every word and phrase in the target sentence, adding more and more weights to transitions that keep appearing in the corpus.<br />
<br />
To quickly illustrate the idea, let's consider the following input and output sentence where each letter represents a word:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">a b c -> w x y z </span><br />
<br />
Without understanding what maps to what let's simply create all possible mappings with the following rules:<br />
- 1:1 - map all single word to all single words in the output.<br />
- 1:2 - map all single words in the input to all two-word phrases in the output<br />
- 2:1 - map all two-word phrases in the input to every single word in the output<br />
- apply the above logic to 2:2, 3:2, 2:3, and 3:3.<br />
These steps are necessary to cater for the differences in fertility in the languages and phrases.<br />
<br />
Applying the above steps to the sample sentence would yield a total of 39 transitions so I was expecting Don Juan graph to be reasonably large. And it was, the graph had a total of 1,349,175 unique edges with 1,576,325 training transitions added, meaning in 227,150 cases an edge received 2 or more reinforcements.<br />
<br />
<h2 style="text-align: left;">
Weighting the edges</h2>
Adding the edges without any further consideration, as in every training simply increasing the weight of the edge by a fixed value, didn't really give usable results, it was too skewed and too noisy.<br />
<br />
To make more sense I had to consider adding the following weighting to the edge:<br />
- the shorter the sentence is, the more relevant the transition is<br />
- the closer the lengths of the phrases are, the more likely the transition is, as in a 1:1 mapping is more likely than a 1:2<br />
- the closer the phrases are to each other in each sentence the more likely the transition<br />
<br />
More formally:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">weight += 1 / (1 + len(source sentence)^2 + diff(sourcePos, destinationPos) + diff(source words count, dest words count))</span><br />
<br />
<h2 style="text-align: left;">
Results</h2>
While there is definitely space for improvement on weights, the results were more than adequate. After generating the sample translation with a weight cut-off I ended up with 20,223 translations. A lot of these were obviously just noise but the more obvious translations were there and (at least somewhat) correct. For instance:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">qué -> ['what', 'what's', 'you']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sí -> ['yes', 'the', 'no']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">no -> ['no', 'i', 'don't']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">la -> ['the', 'you', 'i']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">el -> ['the', 'and', 'i']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">vamos -> ['go', 'let's', 'let's go']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">yo -> ['i', 'i am', 'am']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">yo soy -> ['that's me', 'that's', 'i am']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">estoy -> ['i', 'am', 'am i']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">estoy yo -> ['am i', 'am i not', 'am']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">qué pasa -> ['what's happening', 'what's', 'happening']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">extraño -> ['strange', 'a', 'man']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">es su casa -> ['is his house', 'is his', 'this is his']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">quiero hablar con -> ['i wish to', 'i wish', 'wish to speak']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">vete -> ['go', 'go in', 'in']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">bebamos -> ['lets', 'lets drink', 'drink']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">por qué -> ['why', 'for what', 'for']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">la mesa -> ['the', 'table', 'the table']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">en la mesa -> ['at the table', 'at the', 'part at the']</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: inherit;"><a href="https://gist.githubusercontent.com/adam-ho/2b0715f3b470ba408fe99cda4a8cce8d/raw/0b75c0150a6ae2554e2c6154341e7d9363706956/all_translations_es_en.txt" rel="nofollow" target="_blank">Full Spanish - English translation available here.</a></span><br />
<br />
<br />
<blockquote class="imgur-embed-pub" data-id="a/c0Ydl" lang="en">
<a href="https://imgur.com/c0Ydl"></a></blockquote>
<span style="font-size: x-small;"><i><br /></i></span>
<script async="" charset="utf-8" src="//s.imgur.com/min/embed.js"></script><span style="font-size: x-small;"><i>(A very small subset of the new translation graph, highlighting the most confident ones with red.)</i></span><br />
<span style="font-size: x-small;"><i><br /></i></span>
<br />
<h2 style="text-align: left;">
Limitations</h2>
The used sample was very small which yielded in a very noisy output with a lot of incorrect translations. It's generally a problem with statistical machine translations as they would translate practically anything, but frequently incorrectly without even knowing it.<br />
The language used in Don Juan is somewhat archaic so a lot of modern phrases were not present at all.<br />
Also, the mapping didn't take anything into account that was longer than 3 words so certain phrases were incorrectly mapped because of this chunking.<br />
<br />
<span style="font-size: x-small;"><i>(Dedicated to P. Thanks!)</i></span><br />
<div>
<br /></div>
<h2 style="text-align: left;">
Full source code</h2>
<script src="https://gist.github.com/adam-ho/76dad945a5aac3a5416ec12ee0388c2b.js"></script>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-14885859834028766722017-08-13T18:56:00.000-07:002017-08-13T18:56:03.745-07:00Detecting language features and building machine translation based on sample bilingual corpus<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Representing the language as a directed graph can give us insights on how the language is structured, what are the typical phrases and potentially, if this graph can be matched to another graph in a different language, automatically build up a machine translation graph.<br />
<br />
<h2 style="text-align: left;">
Language as a graph</h2>
<br />
Let's say we analyse the following two simple sentences:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">"The cat is brown."</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">"Don't play with the cat."</span><br />
<br />
These would yield the following directed graph:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbCzUbI5nVs8oy2PQ6WIYtnzUQ3TBDRZEoxWQcJ0WGAcDWMijSKU77BFf_QrZDY4IR6Ulsq367fEzdT5e31Qng2B37VAWaaYrWiWjKMbXdATVUCa8W1DFKWCQgxeqpaJwPJsS5UYz8seI/s1600/the_cat_is_brown.png" imageanchor="1"><img border="0" height="242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbCzUbI5nVs8oy2PQ6WIYtnzUQ3TBDRZEoxWQcJ0WGAcDWMijSKU77BFf_QrZDY4IR6Ulsq367fEzdT5e31Qng2B37VAWaaYrWiWjKMbXdATVUCa8W1DFKWCQgxeqpaJwPJsS5UYz8seI/s320/the_cat_is_brown.png" width="320" /></a><br />
<br />
As we can see the transition from "<span style="font-family: "courier new" , "courier" , monospace;">the</span>" node to the "<span style="font-family: "courier new" , "courier" , monospace;">cat</span>" node has a weight of 2 because in this sample the most likely transition from the word "<span style="font-family: "courier new" , "courier" , monospace;">the</span>" is to the word "<span style="font-family: "courier new" , "courier" , monospace;">cat</span>".<br />
<br />
Given large enough text corpus the typical phrases in a language would become obvious and if we give correct weights to the transitions while removing the noise the graph would actually be meaningful.<br />
<br />
<h2 style="text-align: left;">
Analysing Don Juan</h2>
<br />
The first task was to obtain a large enough homogeneous corpus in preferably two different languages to train the sample. The <a href="https://www.gutenberg.org/files/5201/5201-h/5201-h.htm" rel="nofollow" target="_blank">Gutenberg project</a> is a very good source for quality corpuses and I picked a classic one, Don Juan.<br />
<br />
As the source was in HTML I used BeautifulSoup to parse out the Spanish and English translations and used a simple python script to build the weighted directed graph (source code at the end).<br />
<br />
Having a lot of pairs expressed as "Don -> Juan, 34" wasn't really useful to the human eye, so I used <a href="http://www.graphviz.org/" rel="nofollow" target="_blank">Graphviz</a> to draw the graph from the potential transitions, ignoring the less likely ones. So for instance, if in the corpus I only found "la -> mesa" transition less than 5 times, I ignored it, making the resulting graph more meaningful.<br />
<br />
Graphviz dot tool does take the weights into account, so if a weight is higher it will move the nodes closer to each other, and the results were more than meaningful! The python script, that had no idea what a language is and had no other information just the text yielded surprisingly good insights on the structure of a language.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ27jtJ0GuIdIgV79p0U-2SelPuVTGasUJvRodi1kRQUMFO_OuLibJecbOqAPOXcfHrOzKO0ytQqEPWVC8IijNHycII-vDVSOx9VQCM2JvRdZZkHGyYn-p5eqFR9xbpHpFXZYCfV6UPO0/s1600/g.es.1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="117" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ27jtJ0GuIdIgV79p0U-2SelPuVTGasUJvRodi1kRQUMFO_OuLibJecbOqAPOXcfHrOzKO0ytQqEPWVC8IijNHycII-vDVSOx9VQCM2JvRdZZkHGyYn-p5eqFR9xbpHpFXZYCfV6UPO0/s320/g.es.1.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Crop of Spanish structure - 1</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEIqYjNRma4wVJ-a6rwhnaewB8mE96UXr18ywWYmjpt3iRtSJYsWTW-FtXk1vulH4Xwws0-CcEQ_RekHezUW7LxqSh_JynWyRmBlT1pyVlyM9iU59fRZ63j7yEnoXv4xmQOFtWpTyklOk/s1600/g.es.2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="56" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEIqYjNRma4wVJ-a6rwhnaewB8mE96UXr18ywWYmjpt3iRtSJYsWTW-FtXk1vulH4Xwws0-CcEQ_RekHezUW7LxqSh_JynWyRmBlT1pyVlyM9iU59fRZ63j7yEnoXv4xmQOFtWpTyklOk/s320/g.es.2.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Crop of Spanish structure - 2</td></tr>
</tbody></table>
<blockquote class="imgur-embed-pub" data-id="a/yNrVN" lang="en">
<a href="https://imgur.com/yNrVN"></a></blockquote>
<script async="" charset="utf-8" src="//s.imgur.com/min/embed.js"></script>
<br />
<h2 style="text-align: left;">
Translation between languages</h2>
<br />
The other idea was that if I can find a clear phrase to phrase transition enough times in the corpus I can consider that a valid translation. Unfortunately, the mapping between the source and destination phrases within a sentence is quite difficult, so I cut some corners for this specific exercise: only a full sentence or line to another full sentence or line is considered to be a valid mapping.<br />
<br />
As it was expected, this approach did not uncover too many translations, but the ones it did were actually accurate! Again, all this without any dictionaries or prior knowledge about the source or the destination languages. I used Spanish and English in this case because I happen to speak both so I can verify the correctness of the approach, but I could have used Korean and French and the results would be equally accurate.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggFeWAMTTBJWXtvhUIqiEGtso6HWGeusRtrkXVDv8bikeje2IamkXUiQhLLSHy9ARsK-qGDXo5JClXoMBHV3AM5THbwYXcU3O1T_tCYPOzdre43jqVh3CZ0EibXF7MB6hxaQp1abKEi4k/s1600/g.translate.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="301" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggFeWAMTTBJWXtvhUIqiEGtso6HWGeusRtrkXVDv8bikeje2IamkXUiQhLLSHy9ARsK-qGDXo5JClXoMBHV3AM5THbwYXcU3O1T_tCYPOzdre43jqVh3CZ0EibXF7MB6hxaQp1abKEi4k/s320/g.translate.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Automatic translations found from Spanish to English</td></tr>
</tbody></table>
<br />
<h2 style="text-align: left;">
Further work</h2>
<br />
The next step will be to try to iteratively build up the translation of the phrases, starting from the simple 1:1 word mapping then moving on the 1 & 2 words : 1 & 2 words type of mapping and so on. The idea is that once we understand that both "el" and "la" is mapped to "the", it's easier to understand that the phrase "la mesa" means "the table".<br />
<br />
The first version that I've written yielded too noisy output so it definitely needs further refinements; having said that, it did find many translations that the previous sentence/line level translation did not uncover.<br />
<br />
<h2 style="text-align: left;">
Source code (python, requires <a href="https://www.crummy.com/software/BeautifulSoup/" rel="nofollow" target="_blank">BeautifulSoup</a>)</h2>
</div>
<script src="https://gist.github.com/adam-ho/ea43081905d6a92ab18d9b871b6763f9.js"></script></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-25544336746862081582017-04-06T22:07:00.001-07:002017-04-06T22:07:29.341-07:00Apache Spark - crash course<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="separator" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCbnELEBVSFJWx6mD4KuvM4-ZT-OkatP7g-FYn5Ypz4qU0FFGVHwrQKs5m7LgRmqboC4eWaj7u7UlToKCOFt3mflKsszwch23aCS77_cPI5FqO6YOzLhFuLzB4f2hlWaxMsSYxvFQ_-IM/s320/spark.png" width="320" /></div>
Working with NoSQL databases can be very inconvenient as we lose even the basic tools to get insights on our data. For instance, answering very simple questions like “How many customers bought a specific product?” can be nearly impossible, depending on the data structure we built. Simply put, removing <span style="font-family: "courier new" , "courier" , monospace;">GROUP BY,</span> <span style="font-family: "courier new" , "courier" , monospace;">JOIN,</span><span style="font-family: Times, Times New Roman, serif;"> and </span><span style="font-family: "courier new" , "courier" , monospace;">WHERE</span> operators from a database would render it useless for ad-hoc queries. NoSQL databases have never had these operators in the first place, making them a non trivial choice for ad-hoc data analytics.<br />
<br />
Of course if engineering knew up front what kind of queries will hit the system, they could have denormalized the data to the point that these very specific questions can be answered easily, but ad-hoc queries would be still impossible.<br />
<br />
<h3 style="text-align: left;">
Spark to help</h3>
<br />
Apache Spark is a very interesting concept: it is a distributed data access engine that can work on multiple underlying data stores, providing a consistent API for the developers. For instance, it can run on Hadoop, Cassandra, and MongoDB too; it can even work with some simple shared file systems like NFS. For a full list, check out the <a href="https://spark-packages.org/?q=tags%3A%22Data%20Sources%22" rel="nofollow" target="_blank">Spark data access packages</a>.<br />
<br />
Running on some underlying data store it provides both batch and streaming capabilities.<br />
<br />
Another unique property of Spark is that it tries to keep the intermediate data in memory for the next computations, unlike Hadoop, which is the traditional batch map-reduce framework: write the results to disk, start the next stage. Spark can spill the data to disk when the memory is full, but it will try not to. The intermediate data caching (<span style="font-family: "courier new" , "courier" , monospace;">.cache</span>) can be fine tuned with the <a href="http://spark.apache.org/docs/latest/programming-guide.html#which-storage-level-to-choose" rel="nofollow" target="_blank">storage level parameters</a>.<br />
<br />
<h3 style="text-align: left;">
Streaming operations</h3>
<br />
Spark can respond in a time-window fashion to incoming data, utilising all the computing nodes. This is a typical use case for dealing with large volumes of incoming data that doesn’t need to be stored permanently. Good examples would be webpage impressions, or user clicks. However, Spark Streaming can be utilised for very complex operations too, for instance <a href="https://www.infoq.com/presentations/netflix-recommendation-spark" rel="nofollow" target="_blank">Netflix built their recommendation engine on top of Spark Streaming</a>.<br />
<br />
<h3 style="text-align: left;">
Batch operations</h3>
<br />
Apache Spark provides two basic ways to interact with it in a batch fashion, as in read some existing data, transform it, and save it or display it: the standard Cassandra Context API, and Hive API (for SQL queries). All of these are available both in a standalone app and on the interactive console as well, so we can either write Scala queries, or even standard SQL queries - <span style="font-family: "courier new" , "courier" , monospace;">JOIN</span> and <span style="font-family: "courier new" , "courier" , monospace;">GROUP BY</span> are back!<br />
<br />
Whilst the SQL API is very convenient, it’s important to learn what you can, cannot, or shouldn’t do when running your queries. As you are working on terabytes of data (otherwise why bother with a hard to use NoSQL database in the first place?), some of the queries can exhaust the disk IO, the network, or the CPUs.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4XzK9N0XyTjKugnXAjXxQ4mxqxC9HdsCPMgc1JXVfdAe7j3yWZl3h9rzoHUHsy-B0qUFablEZC1wK3K0C-5UHcusFoiDiQ0-BGlVZDsBnGSMv1h2FPlkL2S4fy-ybrhne7Mg85T1JAQE/s1600/spark-network.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4XzK9N0XyTjKugnXAjXxQ4mxqxC9HdsCPMgc1JXVfdAe7j3yWZl3h9rzoHUHsy-B0qUFablEZC1wK3K0C-5UHcusFoiDiQ0-BGlVZDsBnGSMv1h2FPlkL2S4fy-ybrhne7Mg85T1JAQE/s320/spark-network.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A typical batch operation first needs a lot of CPU, bandwidth, and network packets between the nodes, but at the end of the job when the aggregations happen, it will shuffle a lot of smaller packets with much smaller bandwidth.</td></tr>
</tbody></table>
<br />
<h3 style="text-align: left;">
Spark components</h3>
<br />
<b>Driver</b> - the node we are interacting with, the one that will ask the cluster to execute the Job and get the results back.<br />
<b>Job</b> - the work that needs to be parallelized and broken down on the cluster to Tasks and executed by the Workers.<br />
<b>Master</b> - it will be elected dynamically and responsible for distributing the work between the workers.<br />
<b>Worker</b> - a node that can run the Spark jobs.<br />
<b>Executor</b> - the actual process on the Worker that will execute the Tasks and cache the intermediate results.<br />
<b>Task</b> - a unit of independent work that an Executor can execute.<br />
<br />
<h3 style="text-align: left;">
Starting the interactive shell</h3>
<br />
Either run:<br />
<br />
<pre>SPARK_HOME/bin/spark-shell
</pre>
<br />
Or if you have DataStax Enterprise then:<br />
<br />
<pre>dse spark
</pre>
<br />
For a more complete configuration on how to run Apache Spark on a Cassandra cluster when SSL is enabled on Cassandra:<br />
<br />
<pre>dse
-u cassandraUsername -p cassandraPassword
spark
--conf spark.cassandra.connection.ssl.enabled=true
--conf spark.cassandra.connection.ssl.trustStore.password=javaTrustStorePassword
--conf spark.cassandra.connection.ssl.trustStore.path=/etc/dse/cassandra/.truststore
--total-executor-cores=144
--executor-memory 10g
</pre>
<br />
In the example above we are asking for a total of 144 CPU cores to be allocated to the job across the cluster, and each executor should allocate exactly 10 gigabytes of memory up front. On a cluster with 18 nodes it will preallocate 8 cores and 10g on each box for this specific job. If I start another shell with the same parameters, it will allocate another set of 8 cores and another 10g on each node!<br />
<br />
The default Spark shell uses Scala, so brushing up on Scala will be helpful.<br />
<br />
<h3 style="text-align: left;">
Tracking your jobs</h3>
<br />
There are two important URLs to check to see how the jobs are running.<br />
The master UI is available on<br />
<a href="http://localhost:7080/">http://localhost:7080/</a><br />
It will display resource allocation and currently running tasks.<br />
<br />
To track in details what’s being run on the cluster, have a look at<br />
<a href="http://localhost:4040/jobs">http://localhost:4040/jobs</a><br />
<br />
Also, the Apache Spark progress bar looks a bit unusual, so let’s have a quick look at it:<br />
<br />
<pre>[Stage 0:===================================> (403 + 136) / 601]
</pre>
<br />
Stage shows which stage of the batch job is being run. Some batches will require only a single stage, some will need to keep processing the data further, for instance filter, group, then sort. The latter will have more distinct stages and the cluster must finish a stage completely before moving to another one.<br />
<br />
On the progress bar above the cluster has completed 403 tasks out of 601 total tasks, and 136 cores are actively running a task (out of the 144 that we preallocated previously).<br />
<br />
<h3 style="text-align: left;">
Running Spark Context - Scala crash course</h3>
<br />
The following examples can be executed on the freely available DataStax Spark virtual box and most of them are self explanatory so I haven’t really added more description. The jobs below are mostly operating on RDDs (Resilient Distributed Dataset) and only the results of the transformations are collected back to the driver as an array for instance.<br />
<br />
It’s important to understand that transformations and actions are very different: the transformation is a method that will be executed later (<span style="font-family: "courier new" , "courier" , monospace;">filter(), map()</span>, etc.), when an action (<span style="font-family: "courier new" , "courier" , monospace;">collect(), take(),</span> etc.) is performed on the RDD. Until then, it’s just a definition, so for instance the following first line doesn't actually open the Cassandra table; it will only happen on the second line when the processing pipeline is triggered by the <span style="font-family: Courier New, Courier, monospace;">.take()</span> action.<br />
<br />
<pre class="brush:csharp">val movies = sc.cassandraTable("killr_video", "videos_by_actor")
movies.select("actor_name").as( (name:String) => (name, 1)).
reduceByKey(_ + _).map( e => (e._2, e._1)).sortByKey(false).take(10)
/////////
val moviesTH = movies.select("video_id", "release_year", "title").
as( (id:String, r:Integer, t:String) => (id, t + " " + r)).
where("actor_name = 'Benicio del Toro'")
val moviesJD = movies.select("video_id", "release_year", "title").
as( (id:String, r:Integer, t:String) => (id, t + " " + r)).
where("actor_name = 'Johnny Depp'")
moviesTH.join(moviesJD).mapValues{ case (title1, title2) => title1}.
collect.foreach(println)
/////////
val moviesTH = movies.select("video_id", "release_year", "title").
as( (id:String, r:Integer, t:String) => (id, t + " " + r)).
where("actor_name = 'Benicio del Toro'")
val moviesJD = movies.select("video_id", "release_year", "title").
as( (id:String, r:Integer, t:String) => (id, t + " " + r)).
where("actor_name = 'Johnny Depp'")
moviesJD.intersection(moviesTH).collect.foreach(println)
/////////
val movieRatings = sc.cassandraTable("killr_video", "videos").
select("video_id", "avg_rating").
as( (id:String, r:Option[Float]) => (id, r))
val moviesJD = sc.cassandraTable("killr_video", "videos_by_actor").
where("actor_name = 'Johnny Depp'").
select("video_id", "title").
as( (id:String, t:String) => (id, t))
val ratingsJD = moviesJD.leftOuterJoin(movieRatings).
map{ case (id, (title, rating)) => (rating.get.get, title) }
ratingsJD.sortByKey(false).collect.foreach(println)
## CREATE TABLE killr_video.johnny_ratings(title text PRIMARY KEY, rating float);
ratingsJD.saveToCassandra("killr_video", "johnny_ratings", SomeColumns("rating", "title"))
/////////
val bestvideos = sc.cassandraTable("killr_video", "videos").
select("video_id", "title", "avg_rating").
as( (id:java.util.UUID, t:String, r:Option[Float]) => (id,(t, r.getOrElse(0.0f).asInstanceOf[Float])) ).
sortBy( t => t._2._2, false).take(10)
val actors = sc.cassandraTable("killr_video", "videos_by_actor").
select("video_id", "actor_name").
as( (id:java.util.UUID, t:String) => (id, t))
val bestvideosrdd = sc.parallelize(bestvideos)
bestvideosrdd.join(actors).collect.foreach(println)
/////////
val movieRatings = sc.cassandraTable("killr_video", "videos").
keyBy(row => row.getUUID("video_id"))
val moviesJD = sc.cassandraTable("killr_video", "videos_by_actor").
where("actor_name = 'Johnny Depp'").keyBy(row => row.getUUID("video_id"))
val ratingsJD = moviesJD.leftOuterJoin(movieRatings).
map{ case (id, (rowJD, rowM)) => (rowM.get.getFloat("avg_rating"),rowM.get.getString("title"))}
ratingsJD.sortByKey(false).collect.foreach(println)
/////////
import scala.util.parsing.json._
val obj = JSON.parseFull("{\"a\": 11}")
obj.get.asInstanceOf[Map[String, Any]]("a")
/////////
val movies = sc.cassandraTable("killr_video", "videos").
keyBy( row => row.getInt("release_year") ).
partitionBy(new org.apache.spark.HashPartitioner(2 * sc.defaultParallelism)).cache
movies.partitioner
movies.partitions.size
movies.countByKey.foreach(println)
movies.groupByKey.filterByKey( k => k == 1942).collect.foreach(println)
/////////
sc.cassandraTable("killr_video", "videos_by_actor").
where("actor_name = 'Johnny Depp'").cassandraCount
/////////
sc.cassandraTable("killr_video", "videos_by_actor").
where("actor_name = 'Johnny Depp'").count
/////////
sc.cassandraTable("killr_video", "videos_by_actor").
select("title", "actor_name").
as( (t:String, n:String) => (n, t)).spanByKey.take(10)
sc.cassandraTable("killr_video", "videos_by_actor").
select("title", "actor_name").
as( (t:String, n:String) => (n, t)).groupByKey.take(10)
/////////
case class ActorYear(actor_name: String, release_year: Integer)
val actors2014 =
sc.parallelize(List(ActorYear("Johnny Depp", 2014), ActorYear("Bruce Willis", 2014)))
actors2014.joinWithCassandraTable("killr_video", "videos_by_actor").
on(SomeColumns("actor_name", "release_year")).collect.foreach(println)
/////////
case class Actor(actor_name: String)
val actors =
sc.parallelize(List(Actor("Johnny Depp"), Actor("Bruce Willis"))).
repartitionByCassandraReplica("killr_video", "videos_by_actor")
actors.joinWithCassandraTable("killr_video", "videos_by_actor").takeSample(false, 10)
/////////
import java.io._
val er = sc.cassandraTable("killr_video","videos_by_actor")
val writer = new PrintWriter(new BufferedWriter(new FileWriter("/tmp/piglet2.txt")))
er.select("actor_name", "release_year").
as( (ac:String, y:Int) => Array(ac, y).mkString(",")).
collect.
foreach(writer.println)
writer.flush
writer.close
</pre>
<br />
<h3 style="text-align: left;">
Working with Cassandra SQL / DataFrames</h3>
<br />
The SparkContext interface and Scala is fun to work with, but in some cases it’s much more convenient to write a simple SQL query to achieve the same result (<i>note: sqlContext was called csc in the previous versions of Spark interactive shell</i>). When we are using the Hive API we will mostly work with DataFrames as opposed to the RDDs in the sample above. DFs are more SQL like data structures than the Scala / JVM oriented RDDs, so it's easier to display and filter them.<br />
<br />
<br />
<pre class="brush:csharp">sqlContext.sql("SELECT * FROM killr_video.videos_by_actor").take(10)
/////////
sqlContext.sql("SELECT * FROM killr_video.videos_by_actor").show
/////////
sqlContext.sql("SELECT count(*) FROM killr_video.videos_by_actor where actor_name = 'Johnny Depp'").show
/////////
sqlContext.setKeyspace("killr_video")
sqlContext.sql("SELECT count(*) FROM videos_by_actor where actor_name = 'Johnny Depp'").registerTempTable("jd_movies")
sqlContext.sql("SELECT * FROM jd_movies").show
/////////
sqlContext.sql("SELECT release_year, count(*) FROM killr_video.videos_by_actor group by release_year order by release_year desc").limit(5).show
/////////
sqlContext.sql("SELECT release_year, count(*) FROM killr_video.videos_by_actor group by release_year order by release_year desc").limit(5).explain
/////////
import sqlContext.implicits._
case class ActorYear(actor_name: String, release_year: Integer)
val actors2014df = sc.parallelize(List(ActorYear("Johnny Depp", 2014), ActorYear("Bruce Willis", 2014))).toDF()
val actors2014df = sc.parallelize(List(ActorYear("Johnny Depp", 2014), ActorYear("Bruce Willis", 2014))).toDF("name", "year")
Actors2014df.show
actors2014df.filter("actor_name = 'Bruce Willis'").show
actors2014df.printSchema
actors2014df.schema
actors2014df.dtypes
actors2014df.first.getInt(0)
actors2014df.first.get(1).asInstanceOf[Integer]
</pre>
<br />
<h3 style="text-align: left;">
Further reading</h3>
<div>
<br /></div>
<a href="http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html" rel="nofollow" target="_blank">Apache Spark - all methods with samples </a><br />
<a href="http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/" rel="nofollow" target="_blank">Apache Spark Tuning - part 1</a><br />
<a href="http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/" rel="nofollow" target="_blank">Apache Spark Tuning - part 2</a><br />
<div>
<br /></div>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-19562869007990483842015-09-26T20:19:00.000-07:002024-03-08T12:46:19.038-08:00Why is there more talent in lower ranks than in management?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" height="133" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkxr0dpBnIRYXFq5nElmA7562jy8Vw4BASSI06IJVYTwpLk2poN_eZrVvsdWWpS5eI_G9qsQMvh-Ub68MJwdZUIZdzbu1oh9wFFvkwCkc-89uDBjWL1Ldht4WakUVte2gphrsTp9pErts/s200/lack-of-talent-corporate-ladder.jpg" width="200" /></div>
At generally unmotivated companies, why can you find more motivation and talent in the lowest ranks than in management?<br />
<br />
<b>Not a number game</b><br />
<br />
It obvious that at large, unmotivated organisations there are a lot of hidden gems: talented people at the lowest ranks who somehow just don’t belong there. One of the countless examples would be Nokia where, even when the company went into free-fall and was losing market share by the day, some of the best engineers in the world were still working there – mostly on the lowest rungs of the corporate ladder.<br />
<br />
However, as we start to look up that ladder, we find fewer and fewer of these world-class talents; some might argue that this is just a numbers game. If there are 100 people on one rung of the ladder, then the next rung only has around 10 people, so it’s 10 times less likely to find someone exceptional there. That argument is incorrect. In an ideal setup, the selection criteria for attaining that next level should be 10 times more stringent, therefore we should find a much greater percentage of motivated and talented people on each upper level. We generally don’t.<br />
<br />
<b>The difference that the CEO makes</b><br />
<br />
Before jumping to any conclusions, let’s try to look at the problem from the very top of the ladder. The influence of the CEO – or the lack thereof – has been researched by <a href="http://personalpsychology.com.au/" target="_blank">psychologists</a> since the 1930s. Jeffrey Immelt, the current CEO of General Electric, even argued that in the 1990s the company could have been run by a German shepherd – the job was so easy. His predecessor, Jack Welch, more or less agreed with this statement.<br />
<br />
A study released in 1972, <i>Leadership and Organizational Performance: A Study of Large Corporations</i>, found that the so-called “CEO effect” explained only 14.5 per cent of the company’s profits. Industry effects, on the other hand, accounted for 30 per cent, while company effects such as size and history were responsible for 23 per cent. Of course, some CEO decisions were based on pure chance or luck, so the actual influence is even smaller.<br />
If the overall influence of a CEO is debatable, what deters talent from joining management ranks?<br />
<br />
<b>Reinforcing effect</b><br />
<br />
Two things generally happen that keep unmotivated and untalented managers in place and talented ones away. First, companies are actively canvassing for similarities. As the book Hard Facts, Dangerous Half-truths and Total Nonsense discusses, companies prefer to hire managers who are similar to the other managers in terms of gender, education, thinking, motivation and so on.<br />
<br />
The other reason is that, even if there was a push from upper management to try to get different people with different ideas into management, the existing managers would strongly resist this idea; they would prefer to select people who are similar to them, a preference that’s partly human nature and partly a fear of talent. The pace of work and more extreme ideas talented people bring to a company simply scares them. If they promoted smart and motivated people, it would be only a matter of time before their job would be made redundant. They have to keep up the appearance of importance.<br />
<br />
As James G March discussed in his 1980 book <i>How We Talk and How We Act</i>, “An intelligent manager learns that some of the more effective ways of improving measured performance have little to do with improving product, service, or technology. A system of rewards linked to precise measures is not an incentive to perform well.”<br />
<br />
<b>Upper management and lack of feedback</b><br />
<br />
If a company doesn’t employ 360-degree feedback, it is not just a sign of upper management expressing very little interest in the actual performance of lower management. It also sets up an unidirectional interest: managing up.<br />
<br />
Suddenly, anyone who reports to the manager becomes totally irrelevant, as they are not on the manager’s promotional path in any way. The job of the manager moves from managing people and their careers to pleasing his or her own boss. These managers become too busy and unavailable for anyone reporting to them because subordinates just generate problems and noise, which distracts the manager from pleasing their superiors. Such managers are generally only available to their subordinates if it concerns something that they can re-present to their own managers, so improving their own reputation.<br />
<br />
This, of course, causes frustration in the lower ranks, where people lose control over their growth and career.<br />
<br />
<b>Movement in the lowest ranks</b><br />
<br />
The lack of interest from management usually frustrates talented employees who are trying to achieve something; usually something larger than what they are explicitly asked to do. This in itself is a good enough reason to look around for opportunities in other companies, as talents are not only hard to find but hard to convince to join: smart companies do everything possible to lure talents in, like free meals, extra holidays, team events that are actually interesting … and so on.<br />
<br />
Even if, for some reason, talented people get promoted to managerial level at unmotivated companies they would feel a lack of fulfilment very quickly since, while some of their previous peers were motivated, almost all of their new manager peers are unmotivated and untalented.<br />
<br />
This frustration, of course, never affects less talented workers so they stay because they know there isn't much else for them out there. Also, they see that people like them once in a while get promoted, so there isn’t really any motivation to leave. This eventually leads to contra-selection.<br />
<br />
<b>Smart companies get smarter</b><br />
<br />
For the same reason smart companies promote smart and motivated people into management, and these people tend to promote more and more talented employees, so the whole process is reversed at a smart company. This also means that in the long term the gap between these companies widens: as the unmotivated and untalented company gets duller and duller, the smart and motivated company gets more and more switched on. However, it’s not impossible that there are perfectly “balanced” companies, which on average neither grow more or less motivated.<br />
<br />
This process is accelerated even further by the movement of employees between companies: talented people leave the unmotivated companies because they can’t really grow there and join the motivated ones; less talented workers at smart companies are asked to look for other opportunities so they join the unmotivated companies. It’s a process that widens the gap employee by employee.<br />
This also means that, in the long run, the unmotivated companies will be simply overrun by smart companies. Not only are they not motivated to do anything about the appearance of fierce competition from smart companies, they simply don’t have the talent to fight back. Those people left a long time ago…
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-21008931700776201322015-06-19T20:38:00.001-07:002015-06-22T15:22:11.331-07:00Removing Malware and Bloatware from Lenovo K3 Note (K50-T5)<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8c3mugv8biPo41rWpDnpvTok0eiwVoxyoRnpie6lt86jnI-6ooFmZ_McsPOPPLo-ot6UaBPMmhAw0mBpOvJDs3jEmtuikXI4snDrigs9V_G2m76noK6nltIlqCmpfqpPJ2J3Lj6wBK54/s200/k3note.jpg" width="200" /></div>
I’ve recently purchased a Lenovo K3 Note from Gearbest and it came with preinstalled malware and bloatware. Here are the things I had to do to bring the phone into a usable state.<br />
<br />
<b>Root it</b><br />
<br />
Lot of the malware was preinstalled system apps which cannot be removed without root mode.<br />
<br />
I’ve <a href="http://forum.xda-developers.com/android/development/ufficial-thread-lenovo-k3-note-t3102997" rel="nofollow" target="_blank">rooted using the CWM (ClockworkMod) method</a>, installing the CWM recovery image then installing SuperSU. The <a href="http://forum.xda-developers.com/android/apps-games/one-click-root-tool-android-2-x-5-0-t3107461" rel="nofollow" target="_blank">KingRoot method</a> should be easier but didn’t work for me.<br />
<br />
<b>Remove apps - Bloatware</b><br />
<br />
I’ve used <a href="https://play.google.com/store/apps/details?id=com.keramidas.TitaniumBackup&hl=en" rel="nofollow" target="_blank">Titanium Backup</a> because I wasn’t sure which app would break the phone if removed. I removed the following preinstalled apps (that I remember):<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">BSPTelephonyDevTool 1.0</span><br />
<span style="font-family: Courier New, Courier, monospace;">DU Batter Saver 3.9.9</span><br />
<span style="font-family: Courier New, Courier, monospace;">DU Speed Booster 1.4.0</span><br />
<span style="font-family: Courier New, Courier, monospace;">Mcube</span><br />
<span style="font-family: Courier New, Courier, monospace;">Mobile Assistant 3.0.4.0835</span><br />
<span style="font-family: Courier New, Courier, monospace;">Power Manager 1.1.287.150113.c3631eb</span><br />
<span style="font-family: Courier New, Courier, monospace;">SYNCit 4.0.54</span><br />
<span style="font-family: Courier New, Courier, monospace;"><strike>Theme Center 2.70…</strike> UPDATE: needed for ringtone selection</span><br />
<span style="font-family: Courier New, Courier, monospace;">User Experience 4.4.1</span><br />
<br />
<b>Remove apps – Malware that was display popup ads</b><br />
<br />
The malware that was hijacking my browser was hidden in a modified YouTube app (this <a href="//blog.teamleadnet.com/2015/06/how-to-remove-adware-browser-hijack-or.html" target="_blank">Android Adware was pretty hard to find</a>). I simply uninstalled that using Titanium Backup and installed the correct YouTube app from the Play Store.<br />
<br />
<b>Remove apps – Virus and unofficial apps</b><br />
<br />
The next step was to get rid of the virus, which was hidden in a modified Twitter app. The package name was <span style="font-family: Courier New, Courier, monospace;">com.twiter.android</span>; see how it’s misspelled?<br />
<br />
Unfortunately the infected Twitter app is a system app so cannot be uninstalled, but can be disabled. To do this, I connected the <a href="http://developer.android.com/tools/device.html" rel="nofollow" target="_blank">phone in USB debug mode</a> to the computer and run an adb shell in root mode, and then disabled all unknown or unreliable packages:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./adb shell</span><br />
<span style="font-family: Courier New, Courier, monospace;">su</span><br />
<span style="font-family: Courier New, Courier, monospace;">pm disable com.twiter.android</span><br />
<span style="font-family: Courier New, Courier, monospace;">pm disable com.skymobi.mopoplay.appstore</span><br />
<span style="font-family: Courier New, Courier, monospace;">pm disable com.uc.browser.en</span><br />
<span style="font-family: Courier New, Courier, monospace;">pm disable store.antivirus</span><br />
<span style="font-family: Courier New, Courier, monospace;">pm disable com.cleanmaster.mguard_cn</span><br />
<span style="font-family: Courier New, Courier, monospace;">pm disable com.android.dc</span><br />
<span style="font-family: Courier New, Courier, monospace;">reboot</span><br />
<br />
<b>Things that cannot be uninstalled</b><br />
<br />
I’ve tried to remove the SafeCenter as I have no idea what it does, but it removed the Apps menu from the System menu – so had to reinstall it. Luckily I made a backup with Titanium Backup before deleting it, so everything is back to normal.<br />
<br />
Unfortunately the battery usage breakdown is either gone as well or was never installed in the System menu, but I can’t figure out which app I should reinstall to have it back.<br />
<br />
<b>Viber (or other app) doesn’t work after reboot</b><br />
<br />
Lenovo by default disables auto starting of the apps so Viber won’t start the next time the phone is rebooted and as a result no messages or calls are received. To enable it, go to System Menu, Apps, Viber, Permission Management, and enable Auto-start.<br />
<br />
<br />
Comments? Please let me know!<br />
<div>
<br /></div>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com41tag:blogger.com,1999:blog-8858921836121088131.post-750688343443085232015-06-19T20:07:00.001-07:002015-06-19T20:07:05.999-07:00How to remove adware (browser hijack or pop-up malware) from Android<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXUKRWXQzvz3Ucujx216VVoIaxYAC5MV1N4lcRaPALEXainN9UPnCdotfIdDSCsnvg2VD1kwA1ycMnRMuGfvwWZQK1UXx69DB2gTZQyYQDRX8TAY4lITU4OSkxnipi12BMgqlNJuXoKd0/s200/google-android-malware.jpg" width="177" /></div>
The browser hijack style malware is getting pretty widespread on Android devices, especially the cheaper ones that can be purchased directly from overseas and come with pre-installed malware. Let’s see how to find the adware app that is causing the issue.<br />
<br />
The symptom is usually the same: when the web browser is opened it is redirected to an unknown webpage advertising something totally useless. In some cases a virus infected APK Android package gets downloaded in the hope that it will get installed.<br />
<br />
The app that is causing this issue is waiting in the background for the browser process (either Chrome or Android Browser) to start. When it does it asks it right away to display a webpage.<br />
<br />
<b>The intent</b><br />
<br />
Android natively supports communication between apps, which in itself is very useful, for instance a webpage can be opened from the email client.<br />
The code is pretty straightforward too:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">Uri webpage = Uri.parse(url);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Intent intent = new Intent(Intent.ACTION_VIEW, webpage);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> if (intent.resolveActivity(getPackageManager()) != null) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> startActivity(intent);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> }</span><br />
<br />
The default browser will catch this intent and execute it, open the webpage. As intents can be a good source of problems (for instance, misconfigured intents), the system generally logs them: what was the intent, and which UID sent it.<br />
<br />
The UID is unique on the specific phone for each installed app’s and they are listed in<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">/data/system/packages.xml</span><br />
<br />
<b>Logcat</b><br />
<br />
The first job is to try to consistently reproduce the browser hijacking. Usually force killing the browser, waiting a minute or so and restarting it will trigger the malware to try to hijack it again.<br />
<br />
The second step is to install ADB tool on your computer, enable USB debugging on the phone, plug in to the computer and execute the following command:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./adb logcat > log.txt</span><br />
<br />
This will dump thousand of system log messages to this text file. Now reproduce the browser hijacking and stop the logging.<br />
In the log.txt file the interesting lines look like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">...</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">V/ActivityManager( 729): notify app switch for new activity com.chrome.beta Where 0</span><br />
<span style="font-size: x-small;"><span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">I/ActivityManager( 729): START u0 {act=android.intent.action.VIEW dat=http://global.ymtracking.com/trace?offer_id=100678&aff_id=27742 flg=0x10000000 cmp=com.chrome.beta/com.google.android.apps.chrome.Main} <b>from uid 10035</b> on display 0</span></span><br />
<span style="font-size: x-small;"><span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">D/ActivityManager( 729): notifyAppSwitch resumed: true; pkg:com.chrome.beta</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">...</span><br />
<br />
The middle line is showing that the app under user ID 10035 asked the system to execute a specific URL, a well know malware site.<br />
<br />
If the phone is rooted, run adb again in shell mode to have a look at the app database:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">./adb shell</span><br />
<span style="font-family: Courier New, Courier, monospace;">su</span><br />
<span style="font-family: Courier New, Courier, monospace;">vi /data/system/packages.xml</span><br />
<br />
And find the line that looks like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><package name="... ...version="4003006" <b>sharedUserId="10035"</b>></span><br />
<br />
(if it’s not rooted, there are app that can map from UID to app name in the Play Store).<br />
<br />
In my case it was a modified and virus infected YouTube app, but it can be anything. If it’s not a system app, simply uninstalling it from the apps it enough. If it is a system app and cannot be removed, it’s easy to disable the package so the system cannot execute it any more. Execute the shell again and disable the package:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">pm disable com.package.name.that.is.infected</span><br />
<br />
Reboot the phone and the popups should be gone. If they aren’t, maybe there is another app displaying ads, so repeat the above process and uninstall the other app too.<br />
<br />
<b>Why not just use a virus scanner?</b><br />
<br />
While virus scanners are good at finding actual Android app that contain known viruses, a malware that is hijacking the browser and display popup ads not necessarily known as a virus. Maybe it was custom made for that phone and the wholesaler installed it to earn some extra revenue. Either way, run a virus scanner just to be sure. I’ve found that Nod32 and AVG work well, but only install them from the store! If they come pre-installed, they cannot be trusted.<br />
<br />
Any comments? Please let me know.<br />
<div>
<br /></div>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com26tag:blogger.com,1999:blog-8858921836121088131.post-54982246712731027252015-03-30T22:08:00.000-07:002015-03-30T23:20:06.156-07:00Apple Swift language - crash course & language reference<div dir="ltr" style="text-align: left;" trbidi="on">
Apple recognised that Objective-C is not only not modern but very much not appealing to developers who are used to modern languages. As this can be a barrier to entry it was the right choice to introduce a new language called Swift.<br />
<br />
While the language is similar to most modern languages, it has its own quirks. On the one hand it's somewhat debatable why did Apple invent a brand new language instead of using another existing one, like Google did with Java for Android, on the other hand Swift is actually a well thought and fun to use language. Unlike with Google's own Go, I did not miss anything from the language as I was building my app with it.<br />
<br />
However, it's worth to note that unlike Google's Go, it's not just a language with a compiler, it's a fully supported platform: every OSX and iOS API can be called using Swift, XCode fully supports Swift (eg.: code completion), and most of online Apple Developer documentation is translated to show both Objective-C and Swift samples.<br />
<br />
<b>It's not a managed platform</b><br />
<br />
Jumping on Swift might feel that it's a managed platform, just like Python, Java, or .NET. It is not, there is no garbage collection or anything like that that will magically save us from memory leaks. It is a fully native platform so any memory management is still on us. Obviously Apple helps us with <a href="https://developer.apple.com/library/prerelease/ios/documentation/Swift/Conceptual/Swift_Programming_Language/AutomaticReferenceCounting.html" rel="nofollow" target="_blank">ARC</a> but we still have to understand what is a strong, weak, or unowned reference is, or what is the difference between a class and a struct.<br />
<br />
However, this comes with benefits as well: there is a full interoperability between the Swift and Objective-C code or libraries we use, as they are pretty much compiled to the same binary platform. If we really want to, we can even <a href="https://developer.apple.com/library/ios/documentation/Swift/Conceptual/BuildingCocoaApps/MixandMatch.html#//apple_ref/doc/uid/TP40014216-CH10-XID_77" target="_blank">mix and match</a> the Swift and Objective-C files in the same project; for instance, build the new features in Swift, and maintain the old ones in Objective-C.<br />
<br />
<b>The language basics</b><br />
<br />
As I was reading through the Swift documentation I quickly recognised that it's sometimes too verbose; for developers who haven't really seen other languages maybe. For a "Swift crash course" I collected the most notable language features as simple code samples with comments on what they do. I believe this should be enough to start using the language, but if I missed something please don't hesitate to leave a comment. The left side can be copy-pasted into a Swift playground to try it out.<br />
<br />
<b>The bad</b><br />
<br />
With any new platforms, Swift comes with its own problems. The biggest problems I've seen are:<br />
<br />
<ul style="text-align: left;">
<li>XCode instability - it frequently crashes with Swift source code. </li>
<li>Compiler issues - I've seen a compiler problem where I had to modify to source code as the only workaround.</li>
<li>The language changes quite a bit - since it's release it had two major not-backward compatible updates already.</li>
</ul>
<br />
<br />
<br />
<table style="font-size: x-small;">
<tbody>
<tr>
<td style="vertical-align: top;"><pre class="brush:csharp">//
// Variables, constants
//
var str = "Adam"
var num = 11
let comb = str + String(num)
var optionalName : String?
var eventuallyAssigned : String!
optionalName = "Adam"
if let name = optionalName {
println(name.endIndex)
}
let notOptionalName = optionalName!
eventuallyAssigned = "Adam"
println(optionalName?.endIndex)
println(optionalName!.endIndex)
println(eventuallyAssigned.endIndex)
//
// String formatting
//
println("Formatted \(str + String(num)) :)")
println("Formatted \(str)\(num) :)")
//
// Dictionary
//
var dict = [String : Float]()
dict["Adam"] = 11
println(dict["Adam"])
//
// Loops
//
let numbers = [12,3,12,23,43,53]
var sum = 0
for number in numbers {
sum += number
}
println(sum)
for var i = 0; i < 4; i++ {
println(i)
}
for i in 0..<4 {
println(i)
}
for i in 0...4 {
println(i)
}
//
// Methods, method pointers
//
func greet(name : String, day : String) ->
String {
return "Good \(day) \(name)"
}
println(greet("Adam", "evening"))
func minmax(numbers : [Int]) ->
(min : Int, max : Int) {
var min = numbers[0]
var max = numbers[0]
for number in numbers {
if number < min {
min = number
}
if number > max {
max = number
}
}
return (min, max)
}
func avg(numbers : Int...) -> Double {
var avg = 0.0
var cnt = 0.0
for number in numbers {
avg += Double(number)
cnt++
}
return avg / cnt
}
var mm = minmax([4,3,1,2,6,8])
println(mm.max)
println(mm.0)
println(avg(4,65,7,2,1))
func innerFunc(n : Int) -> Int {
func addFive(m : Int) -> Int {
return m + 5
}
return addFive(n)
}
println(innerFunc(10))
func makeIncrementer(toAdd : Int) ->
(Int -> Int) {
func adder(num : Int) -> Int {
return num + toAdd;
}
return adder
}
let fiveAdder = makeIncrementer(5)
fiveAdder(10)
func myfilter(numbers : [Int], predicate:
Int -> Bool) -> [Int] {
var arr = [Int]()
for number in numbers {
if predicate(number) {
arr.append(number)
}
}
return arr
}
func pred(m : Int) -> Bool {
return m < 5
}
myfilter([1,2,3,4,5,6], pred)
//
// Closures (lambdas)
//
myfilter([1,2,3,4,5,6], {(m : Int) -> Bool in
return m < 5})
let tran = [1,2,3].map({(m :Int) ->
String in String(2*m) + ":)"})
println(tran)
let tran2 = [1,2,3].map({ "a" + String($0 * 2) })
println(tran2)
let r1 = [1,2,3].reduce(0) { $0 + $1 }
println(r1)
let r2 = [1,2,3,4].reduce("= ",
combine: {(s :String, i: Int) ->
String in s + String(i)})
println(r2)
let f1 = [1,2,3].filter{ $0 < 3 }
println(f1)
//
// Classes
//
class Person {
let name : String
let age : Int
// can be nil
weak var child : Person?
// cannot be nil
// unowned var parent : Person
init(name : String, age : Int = 18) {
self.name = name
self.age = age
}
func print(){
println("Hello")
}
deinit {
println("removed person \(name)")
}
}
class Employee : Person {
let salary : Int
var selfRank : Int = 0
init(name: String, age: Int, salary : Int){
self.salary = salary
super.init(name: name, age: age)
}
var rank : Int {
get {
return selfRank;
}
set {
selfRank = newValue
}
}
func selfReferenceWatchOut() {
// Closures are reference types,
// may hold strong reference to self
var closure = {
// Capture list - capturing self,
// no strong ref to self
[unowned self](a: Int) -> Int in
return self.rank
}
}
deinit {
println("removed employee \(name)")
}
}
var p = Person(name: "Adam", age: 11)
var e =
Employee(name: "Adam", age: 11, salary: 12)
e.rank = 11
//
// Generics
//
func repeat<T>(item : T, times : Int) -> [T] {
var arr = [T]()
for i in 0..<times {
arr.append(item)
}
return arr
}
repeat("Adam", 3)
repeat(true, 3)
//
// Protocols (interfaces), and extentions
//
protocol Printer {
func toString() -> String
}
class Printable : Printer {
func toString() -> String {
return "Just implement protocol"
}
}
extension Int : Printer {
func toString() -> String {
return "Or add an extension to \(self)"
}
}
7.toString()
</pre>
</td>
<td style="vertical-align: top;"><pre class="brush:csharp">//
//
//
String variable, which cannot be nil.
Int variable, which cannot be nil.
String constant. Cannot be changed.
Nillable String variable.
Auto unwrapped, nillable variable.
Assigning a variable.
Unwrapping a potentially nil variable.
Only executed if variable wasn't nil.
Unwrap to a constant. If nil, breaks.
Assigning a variable.
Optionally reads endIndex, if var is not nil.
Reads endIndex. If var is nil, breaks.
Auto unwrapped. If nil, breaks.
//
//
//
Prints "Formatted Adam11 :)".
Prints "Formatted Adam11 :)".
//
//
//
Creates String->Float dictionary.
Stores 11 under key "Adam".
Prints 11. If it was nil, prints nil.
//
//
//
Constant array of Ints.
For-each loop.
Prints 146.
Classical for cycle (0,1,2,3)
Shorter for cycle (0,1,2,3)
For cycle inclusive upper bound (0,1,2,3,4)
//
//
//
Method taking String, String, returning String.
Executing and printing method result.
"Good evening Adam".
Method returning two named results.
Returning named results.
Method accepting list of Int arguments.
Holds a tuple of 1 and 8.
Prints 8.
Prints 1.
Executing method with list of arguments.
Declaring inner method within method.
Prints 15.
A method that takes an Int and returns another method
that takes an Int and returns an Int.
Adder is a method.
FiveAdder is a method that takes an Int and returns an Int.
It's 15.
A method that takes a list of numbers and another method
as a parameter.
If the result of the predicate is true,
appends to the new, filtered list.
Returns the new filtered list.
[1,2,3,4]
//
//
//
Execute the filter with a locally defined closure.
[1,2,3,4]
Execute the map function on the array with a local closure
and named parameters.
["2:)", "4:)", "6:")]
Execute the map function on the array with a local closure
and unnamed parameters.
[a2, a4, a6]
Execute the reduce function on the array with local closure
and unnamed parameters.
6.
Execute the reduce function on the array with local closure
and named parameters.
"= 1234"
Executes the filter method on the array with local closure
and unnamed parameters.
[1,2]
//
//
//
Constant, must be initialized in constructor.
Constant, must be initialized in constructor.
Weak reference to itself, can be nil.
Unowned (weak) reference to itself, which couldn't be nil.
Age default parameter is 18.
Destructor.
Inheritance.
selfRank is initialized before constructor.
Initialize parent constructor.
Property with getter and setter methods.
Without the unowned capture list it would be a
strong self reference, and would memory leak.
Any closure that uses "self" has to maintain a
capture list to avoid memory leaks.
Destructor.
Create a Person object.
Create an Employee object.
//
//
//
Generic method that accepts a T typed item and an Int.
Create a dynamic array of type T.
["Adam", "Adam", "Adam"]
[true, true, true]
//
//
//
Protocol (interface) definition.
Implement a protocol.
Adding an extension to another closed class.
Executing the extension on the class.
"Or add an extension to 7"
</pre>
</td>
</tr>
</tbody></table>
</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com1tag:blogger.com,1999:blog-8858921836121088131.post-58425308072627720382014-07-02T03:34:00.001-07:002014-07-02T03:34:36.180-07:00Taming Zookeeper - introduction and “how to” for Java and Python<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQrT50C_Vir8Qy5ul4ELJHhED4Gmp3v3no-toUCeQQ2_Z0OwMwog8RKJ6qW9h9Cib23DBC_ikgKkewqUqaT6jQp0AXBs7p9AypDWfdb_51NG0QnFXvreFhBzQ6O1kkZQAOj2pRPBWZGSg/s1600/zookeeper.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQrT50C_Vir8Qy5ul4ELJHhED4Gmp3v3no-toUCeQQ2_Z0OwMwog8RKJ6qW9h9Cib23DBC_ikgKkewqUqaT6jQp0AXBs7p9AypDWfdb_51NG0QnFXvreFhBzQ6O1kkZQAOj2pRPBWZGSg/s1600/zookeeper.png" height="200" width="113" /></a></div>
Zookeeper is one of the most versatile projects out there but still somewhat underutilized and feared; however, almost every software project could benefit from using it as a distributed, fault tolerant, hierarchical data storage.<br />
<br />
<b>What really is Zookeeper?</b><br />
<br />
Zookeeper is a free and open source, distributed, fault tolerant, hierarchical value storage - pretty much like a network share that stores small files in directories and subdirectories. These “files”, which are called nodes in Zookeeper, are small, so the platform is not really a BLOB storage (actually the size of a node is limited to 1mb).<br />
<br />
Storing and reading these nodes are extremely fast, so any information can be stored in Zookeeper without seconds thoughts – active keys in a cache, currently active sessions, list of active servers, and so on.<br />
<br />
A Zookeeper cluster can consist of multiple servers and there is no predefined master in this topology – the platform will make sure that the storage and retrieval of the nodes are synchronized properly.<br />
<br />
Moreover, Zookeeper guarantees consistency, which means that it doesn’t matter which Zookeeper server the client is talking to, the returned values are always consistent. However, it’s good to keep in mind that consistency is reached on the order of tens of seconds scale - usually it’s much faster but Zookeeper doesn’t consider the service to be down within that time period.<br />
<br />
<b>Cross platform</b><br />
<br />
The core of the Zookeeper server is written in pure Java and consists only of 190 classes, and about 42k lines of code. It is reasonably small for such a system and as the code is nicely written it is far from hard to read it and understand the internals of the system.<br />
<br />
Zookeeper uses a protocol name “Jute” to communicate with the clients, which is somewhat similar to Protobuf that was designed by Google. Interestingly the Jute compiler, which can turn the zookeeper.jute definition into the Java classes, is included in the Zookeeper source code too.<br />
<br />
The protocol is quite simple, so even if there is no direct compilation to other platforms, sending the required data across the wire should be straightforward on most systems. For instance, the Python Kazoo implementation simply recreated the required bindings manually.<br />
<br />
<b>Ephemeral Nodes</b><br />
<br />
One of the most interesting features of Zookeeper is called Ephemeral Node. While permanent nodes are stored forever, an Ephemeral Node will be immediately deleted when the client who created it disconnects from the server. It is one of the best ways to keep track of actual “live” data, like which services or servers are up and running, who are online on our site, or who joined/left a chat room.<br />
<br />
<b>Storage and cleanup</b><br />
<br />
Zookeeper stores the data on disk, so even if the server is restarted, all nodes will available again. As the system is more optimized for performance than storage, cleaning old and deleted data is not part of the Zookeeper server’s normal job.<br />
<br />
Earlier versions required a scheduled job to be run to purge old data, newer versions can do it periodically, but not enabled by default. To enable purging the old data from log files, add/uncomment the following in the zoo.cfg file:<br />
<br />
<pre># keeps the 3 most recent snapshots of the data
autopurge.snapRetainCount=3
# runs every 1 hour
autopurge.purgeInterval=1
</pre>
<br />
<b>Watching data changes</b><br />
<br />
Zookeeper can notify the clients if a watched node has been changed, but every watcher fire only once. If we need to keep watching the changes, we need to create a new watcher; however, it is important to keep in mind that while we will receive the latest change to the node, we may not receive all intermittent changes when we are re-creating the watch, or temporarily get disconnected from the server.<br />
<br />
<b>Starting the server</b><br />
<br />
For development purposes it’s really easy to start using Zookeeper: after downloading the binaries, simply run<br />
<pre>./bin/zkServer.sh start-foreground
</pre>
In production mode it might be a good idea to use it as a service or run though supervisord to make sure it is restarted if it crashes for any reason.<br />
<br />
<b>Java client</b><br />
<br />
The Curator framework provides high-level access for accessing the Zookeeper server. Even though the protocol is simple so it could be quickly re-implemented, the frameworks typically give extra features like automatic reconnection to the server with a predefined exponential back off time.<br />
<br />
To use the Curator framework, simply add the following dependency to the Maven pom.xml file:<br />
<br />
<pre><dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>2.5.0</version>
</dependency>
</pre>
<br />
To read string data from an existing node, only the following is required:<br />
<br />
<pre class="brush:csharp">final RetryPolicy retryPolicy =
new ExponentialBackoffRetry(BASE_SLEEP_TIME_MS, MAX_RETRIES);
final CuratorFramework client =
CuratorFrameworkFactory.newClient(CONNECTION_STRING, retryPolicy);
client.start();
final GetDataBuilder getDataBuilder = client.getData();
final byte[] configBytes = getDataBuilder.forPath(CONFIG_PATH);
final String config = new String(configBytes);
System.out.println("Data: " + config);
</pre>
<br />
For a full fledged demo with watchers, deletion, and Zookeeper ACLs, check out the <a href="https://github.com/adam-ho/misc/tree/master/zookeeperClient.java" rel="nofollow" target="_blank">java source code here</a>.<br />
<br />
<b>Python client</b><br />
<br />
The easiest way to access Zookeeper is to use the Kazoo client library, which is written purely in Python. It can be installed system wide or just used as a module in a solution and deployed as part of the application.<br />
<br />
The steps are very similar to the Java client, so we need to create and start a connection and read the data we are looking for:<br />
<pre>from kazoo.client import KazooClient
zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()
print('Data in node: %s' % zk.get('/rabbit/config')[0])
</pre>
As Kazoo is a high level framework, we do not need to recreate the watcher after every change, it will stay active.<br />
<br />
For the full source code with an embedded Kazoo client, check out the <a href="https://github.com/adam-ho/misc/tree/master/zookeeperClient.python" rel="nofollow" target="_blank">python source code here</a>.<br />
<br />
<b>Performance</b><br />
<br />
The Zookeeper server are surprisingly fast, reaching above 20k operations / second on a singe server setup with 2 cores and 10 simulated clients. The average latency is typically less than 1 millisecond so in most circumstances Zookeeper wont’ be the bottleneck in the system.<br />
<br />
For a detailed performance review of the system under different configurations (cores and machines), <a href="http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview" rel="nofollow" target="_blank">check out this Zookeeper performance article</a>.<br />
<br />
<b>Conclusion</b><br />
<br />
Even though Zookeeper is usually used as part of a larger software package like Apache Hadoop or Apache Storm, it is a great standalone product.<br />
<br />
However, unfortunately it is a little bit mystified with articles going in great depth of the election algorithm of the server nodes, which doesn’t really help to understand how to and when to use Zookeeper.<br />
<br />
Storing simple hierarchical information in a fault tolerant, distributed fashion with the live tracking capability of ephemeral nodes make Zookeeper a really handy tool across a lot of different software projects. Give it a go, <a href="http://zookeeper.apache.org/releases.html#download" rel="nofollow" target="_blank">get Zookeeper binaries</a>!</div>Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com1tag:blogger.com,1999:blog-8858921836121088131.post-23617285464141560452014-06-06T18:36:00.002-07:002014-06-12T14:20:34.862-07:00Beating the binary search algorithm – interpolation search, galloping search<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Binary search is one of the simplest yet most efficient algorithms out there for looking up data in sorted arrays. The question is, can it be beaten by more sophisticated methods? Let’s have a look at the alternatives.<br />
<br />
In some cases hashing the whole dataset is not feasible or the search needs to find the location as well as the data itself. In these cases the <span style="font-family: Courier New, Courier, monospace;">O(1)</span> runtime cannot be achieved with hash tables, but <span style="font-family: Courier New, Courier, monospace;">O(log(n)) </span>worst case runtime is generally available on sorted arrays with different divide and conquer approaches.<br />
<br />
Before jumping to conclusions, it is worth to note that there are a lot of ways to “beat” an algorithm: required space, required runtime, and required accesses to the underlying data structure can be different priorities. For the following runtime and comparison tests different random arrays between 10,000 and 81,920,000 items were created with 4 byte integer elements. The keys were evenly distributed with an average increment of 5 between them.<br />
<br />
<b>Binary search</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvSjSb33zc1T1Nry-i6kg9XW4GoQ7HKk1-sy3M_zUj2UTScGSGgSvsw1cahAMGkUjL5EY1hf-5INp7ttinjAp-sOHzk07F_8IIYJrekmLJElxPfMTdjknG_4CZVpCICFRZES3d8Ivs2xU/s1600/binary.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvSjSb33zc1T1Nry-i6kg9XW4GoQ7HKk1-sy3M_zUj2UTScGSGgSvsw1cahAMGkUjL5EY1hf-5INp7ttinjAp-sOHzk07F_8IIYJrekmLJElxPfMTdjknG_4CZVpCICFRZES3d8Ivs2xU/s1600/binary.png" /></a></div>
The binary search is a guaranteed runtime algorithm, whereas the search space is always halved at each step. Searching for a specific item in an array guaranteed to finish in <span style="font-family: Courier New, Courier, monospace;">O(log(n))</span> time, and if the middle point was selected luckily even sooner. It means that an array with 81,920,000 elements only needs 27 or less iterations to find the element’s location in the array.<br />
Because of the random jumps of the binary search, this algorithm is not cache friendly so some fine tuned versions would switch back to linear search as long as the size of the search space is less than a specified value (64 or less typically). However, this final size is very much architecture dependent, so most frameworks don’t have this optimization.<br />
<br />
<b>Galloping search; galloping search with binary search fallback</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxQ7W17jYfcxGUija4qiIZ5sVBBvGdSFQgZxV0YUfJ927xm7Wn5A_1G6JmtSS35nBZUEHW7ovCa6bOGAV0Y4-_inR9d_m8KD36oEPwfCkVwJL1GuAOF4-xXtk4mQ4Z7olIYSDvOTMjRUg/s1600/gallop.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxQ7W17jYfcxGUija4qiIZ5sVBBvGdSFQgZxV0YUfJ927xm7Wn5A_1G6JmtSS35nBZUEHW7ovCa6bOGAV0Y4-_inR9d_m8KD36oEPwfCkVwJL1GuAOF4-xXtk4mQ4Z7olIYSDvOTMjRUg/s1600/gallop.png" /></a></div>
If the length of the array is unknown for some reason, the galloping search can identify the initial range of the search scope. This algorithm starts at the first element and keeps doubling the upper limit of the search range until the value there is larger than the searched key. After this, depending on the implementation, the search either falls back to a standard binary search on the selected range, or restarts another galloping search. The former one guarantees an <span style="font-family: Courier New, Courier, monospace;">O(log(n))</span> runtime, the latter one is closer to <span style="font-family: Courier New, Courier, monospace;">O(n)</span> runtime.<br />
Galloping search is efficient if we expect to find the element closer to the beginning of the array.<br />
<br />
<b>Sampling search</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9yTVdXc25V0gmXnZP151ovODLjDLaQMwloKeksTJAeGneo3V2K6QMon7lUIuUUMcccwF86Y01c2TXXeDmWvH-7wsWAwd1uyw_-2D4eNo9F5Lxfp0ZnMQDeBzeTbWrBTsMNwv5H-juMKo/s1600/sampling.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9yTVdXc25V0gmXnZP151ovODLjDLaQMwloKeksTJAeGneo3V2K6QMon7lUIuUUMcccwF86Y01c2TXXeDmWvH-7wsWAwd1uyw_-2D4eNo9F5Lxfp0ZnMQDeBzeTbWrBTsMNwv5H-juMKo/s1600/sampling.png" /></a></div>
The sampling search is somewhat similar to the binary search but takes several samples across the array before deciding which region to focus on. As a final step, if the range is small enough, it falls back to a standard binary search to identify the exact location of the element.<br />
The theory is quite interesting but in practice the algorithm doesn’t perform too well.<br />
<br />
<b>Interpolation search; interpolation search with sequential fallback</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2CnOCpunsXn0OOapSCuk1e16fT3_ignRwtVru27c2YNyF0H_3JWOo4qxS9tJoSbyqwF0ARZbEqZkClTLfpYGlNyYrlEyPnslVN6VOa-BYEZI5ODukH7Iu9mcIgUYf3AsZ5sDirlG3b9g/s1600/interpolation.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2CnOCpunsXn0OOapSCuk1e16fT3_ignRwtVru27c2YNyF0H_3JWOo4qxS9tJoSbyqwF0ARZbEqZkClTLfpYGlNyYrlEyPnslVN6VOa-BYEZI5ODukH7Iu9mcIgUYf3AsZ5sDirlG3b9g/s1600/interpolation.png" /></a></div>
The interpolation search supposed to be the “smartest” among the tested algorithms. It resembles the way humans are using phonebooks, as it tries to guess the location of the element by assuming that the elements are evenly distributed in the array.<br />
As a first step it samples the beginning and the end of the search space and then guesses the element’s location. It keeps repeating this step until the element is found. If the guesses are accurate, the number of comparisons can be around<span style="font-family: Courier New, Courier, monospace;"> O(log(log(n))</span>, runtime around <span style="font-family: Courier New, Courier, monospace;">O(log(n))</span>, but unlucky guesses easily push it up to <span style="font-family: Courier New, Courier, monospace;">O(n)</span>.<br />
The smarter version switches back to linear search as soon as the guessed location of the element is presumably close to the final location. As every iteration is computationally expensive compared to binary search, falling back to linear search as the last step can easily outperform the complex calculations needed to guess the elements location on a short (around 10 elements) region.<br />
One of the big confusions around interpolation search is that the <span style="font-family: Courier New, Courier, monospace;">O(log(log(n))</span> number of comparisons may yield <span style="font-family: Courier New, Courier, monospace;">O(log(log(n))</span> runtime. This is not the case, as there is a big tradeoff between storage access time versus CPU time needed to calculate the next guess. If the data is huge and storage access time is significant, like on an actual disk, interpolation search will easily beat binary search. However, as the tests show, if access time is very short, as in RAM, it may not yield any benefit at all.<br />
<br />
<b>Test results</b><br />
<br />
All the code was hand written for the tests in Java (source code at the end); each test was run 10 times on the same array; the arrays were random, in memory integer arrays.<br />
<br />
The interpolation search fell back to linear search if the assumed distance was 10 or fewer items, while the sampling search took 20 samples across the search space before deciding where to continue the search. Also, when the key space was less than 2000 items, it fell back to standard binary search.<br />
<br />
As a reference, Java’s default Arrays.binarySearch was added to compare its runtime to the custom implementations.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsCYRYHDl91eakBDnYjSawvv3rlHkDNJcaoWULSRyhLV88I9GMvcAV4YocvwRgtBrhANjvgFYHGT6W8UmKNOuPsN_0rso3EUKmqCxu6MAmbPZUNpz1q5nJvFlf0opL1qHseY9r2dm95bw/s1600/SearchTime.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsCYRYHDl91eakBDnYjSawvv3rlHkDNJcaoWULSRyhLV88I9GMvcAV4YocvwRgtBrhANjvgFYHGT6W8UmKNOuPsN_0rso3EUKmqCxu6MAmbPZUNpz1q5nJvFlf0opL1qHseY9r2dm95bw/s1600/SearchTime.png" height="195" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Average search time / element, given the array size</td></tr>
</tbody></table>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioqQqexSarTlXfCvuKCRYLbUP2pKaS8tMz98Ge9zBuSGBSW4wiXwT74IjgjMBlYWqgLv4tEtqYJpvvmmBMIApx-SSAi_aN7iI2VyFRfrq6o-XltNU1kviNVwF_8nDvmPUuefB7qe-iM7I/s1600/Comparisions.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioqQqexSarTlXfCvuKCRYLbUP2pKaS8tMz98Ge9zBuSGBSW4wiXwT74IjgjMBlYWqgLv4tEtqYJpvvmmBMIApx-SSAi_aN7iI2VyFRfrq6o-XltNU1kviNVwF_8nDvmPUuefB7qe-iM7I/s1600/Comparisions.png" height="183" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Average comparisons / search, given the array size</td></tr>
</tbody></table>
<br />
Despite of the high expectations for interpolation search, the actual runtime did not really beat the default binary search. When storage access time is long, a combination of some kind of hashing and B+ tree probably would be a better choice, but it is worth to note that on uniformly distributed arrays the interpolation search combined with sequential search always beats the binary search on the number of comparisons required. It’s also interesting how efficient the platform’s binary search was, so for most cases it’s probably not worth it to replace it with something more sophisticated.<br />
<br />
<b>Raw data – average runtime per search</b><br />
<br />
<table class="tableizer-table">
<tbody>
<tr class="tableizer-firstrow"><th><span style="font-size: x-small;">Size</span></th><th><span style="font-size: x-small;">Arrays.</span><br />
<span style="font-size: x-small;">binarySearch</span></th><th><span style="font-size: x-small;">Interpolation</span><br />
<span style="font-size: x-small;">+Seq</span></th><th><span style="font-size: x-small;">Interpolation</span></th><th><span style="font-size: x-small;">Sampling</span></th><th><span style="font-size: x-small;">Binary</span></th><th><span style="font-size: x-small;">Gallop</span></th><th><span style="font-size: x-small;">Gallop</span><br />
<span style="font-size: x-small;">+Binary</span></th></tr>
<tr><td><span style="font-size: x-small;">10,000</span></td><td><span style="font-size: x-small;">1.50E-04 ms</span></td><td><span style="font-size: x-small;">1.60E-04 ms</span></td><td><span style="font-size: x-small;">2.50E-04 ms</span></td><td><span style="font-size: x-small;">3.20E-04 ms</span></td><td><span style="font-size: x-small;">5.00E-05 ms</span></td><td><span style="font-size: x-small;">1.50E-04 ms</span></td><td><span style="font-size: x-small;">1.00E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">20,000</span></td><td><span style="font-size: x-small;">5.00E-05 ms</span></td><td><span style="font-size: x-small;">5.50E-05 ms</span></td><td><span style="font-size: x-small;">1.05E-04 ms</span></td><td><span style="font-size: x-small;">2.35E-04 ms</span></td><td><span style="font-size: x-small;">7.00E-05 ms</span></td><td><span style="font-size: x-small;">1.15E-04 ms</span></td><td><span style="font-size: x-small;">6.50E-05 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">40,000</span></td><td><span style="font-size: x-small;">4.75E-05 ms</span></td><td><span style="font-size: x-small;">5.00E-05 ms</span></td><td><span style="font-size: x-small;">9.00E-05 ms</span></td><td><span style="font-size: x-small;">1.30E-04 ms</span></td><td><span style="font-size: x-small;">5.25E-05 ms</span></td><td><span style="font-size: x-small;">1.33E-04 ms</span></td><td><span style="font-size: x-small;">8.75E-05 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">80,000</span></td><td><span style="font-size: x-small;">4.88E-05 ms</span></td><td><span style="font-size: x-small;">5.88E-05 ms</span></td><td><span style="font-size: x-small;">9.88E-05 ms</span></td><td><span style="font-size: x-small;">1.95E-04 ms</span></td><td><span style="font-size: x-small;">6.38E-05 ms</span></td><td><span style="font-size: x-small;">1.53E-04 ms</span></td><td><span style="font-size: x-small;">9.00E-05 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">160,000</span></td><td><span style="font-size: x-small;">5.25E-05 ms</span></td><td><span style="font-size: x-small;">5.94E-05 ms</span></td><td><span style="font-size: x-small;">1.01E-04 ms</span></td><td><span style="font-size: x-small;">2.53E-04 ms</span></td><td><span style="font-size: x-small;">6.56E-05 ms</span></td><td><span style="font-size: x-small;">1.81E-04 ms</span></td><td><span style="font-size: x-small;">9.38E-05 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">320,000</span></td><td><span style="font-size: x-small;">5.16E-05 ms</span></td><td><span style="font-size: x-small;">6.13E-05 ms</span></td><td><span style="font-size: x-small;">1.22E-04 ms</span></td><td><span style="font-size: x-small;">2.19E-04 ms</span></td><td><span style="font-size: x-small;">6.31E-05 ms</span></td><td><span style="font-size: x-small;">2.45E-04 ms</span></td><td><span style="font-size: x-small;">1.04E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">640,000</span></td><td><span style="font-size: x-small;">5.30E-05 ms</span></td><td><span style="font-size: x-small;">6.06E-05 ms</span></td><td><span style="font-size: x-small;">9.61E-05 ms</span></td><td><span style="font-size: x-small;">2.12E-04 ms</span></td><td><span style="font-size: x-small;">7.27E-05 ms</span></td><td><span style="font-size: x-small;">2.31E-04 ms</span></td><td><span style="font-size: x-small;">1.16E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">1,280,000</span></td><td><span style="font-size: x-small;">5.39E-05 ms</span></td><td><span style="font-size: x-small;">6.06E-05 ms</span></td><td><span style="font-size: x-small;">9.72E-05 ms</span></td><td><span style="font-size: x-small;">2.59E-04 ms</span></td><td><span style="font-size: x-small;">7.52E-05 ms</span></td><td><span style="font-size: x-small;">2.72E-04 ms</span></td><td><span style="font-size: x-small;">1.18E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">2,560,000</span></td><td><span style="font-size: x-small;">5.53E-05 ms</span></td><td><span style="font-size: x-small;">6.40E-05 ms</span></td><td><span style="font-size: x-small;">1.11E-04 ms</span></td><td><span style="font-size: x-small;">2.57E-04 ms</span></td><td><span style="font-size: x-small;">7.37E-05 ms</span></td><td><span style="font-size: x-small;">2.75E-04 ms</span></td><td><span style="font-size: x-small;">1.05E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">5,120,000</span></td><td><span style="font-size: x-small;">5.53E-05 ms</span></td><td><span style="font-size: x-small;">6.30E-05 ms</span></td><td><span style="font-size: x-small;">1.26E-04 ms</span></td><td><span style="font-size: x-small;">2.69E-04 ms</span></td><td><span style="font-size: x-small;">7.66E-05 ms</span></td><td><span style="font-size: x-small;">3.32E-04 ms</span></td><td><span style="font-size: x-small;">1.18E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">10,240,000</span></td><td><span style="font-size: x-small;">5.66E-05 ms</span></td><td><span style="font-size: x-small;">6.59E-05 ms</span></td><td><span style="font-size: x-small;">1.22E-04 ms</span></td><td><span style="font-size: x-small;">2.92E-04 ms</span></td><td><span style="font-size: x-small;">8.07E-05 ms</span></td><td><span style="font-size: x-small;">4.27E-04 ms</span></td><td><span style="font-size: x-small;">1.42E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">20,480,000</span></td><td><span style="font-size: x-small;">5.95E-05 ms</span></td><td><span style="font-size: x-small;">6.54E-05 ms</span></td><td><span style="font-size: x-small;">1.18E-04 ms</span></td><td><span style="font-size: x-small;">3.50E-04 ms</span></td><td><span style="font-size: x-small;">8.31E-05 ms</span></td><td><span style="font-size: x-small;">4.88E-04 ms</span></td><td><span style="font-size: x-small;">1.49E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">40,960,000</span></td><td><span style="font-size: x-small;">5.87E-05 ms</span></td><td><span style="font-size: x-small;">6.58E-05 ms</span></td><td><span style="font-size: x-small;">1.15E-04 ms</span></td><td><span style="font-size: x-small;">3.76E-04 ms</span></td><td><span style="font-size: x-small;">8.59E-05 ms</span></td><td><span style="font-size: x-small;">5.72E-04 ms</span></td><td><span style="font-size: x-small;">1.75E-04 ms</span></td></tr>
<tr><td><span style="font-size: x-small;">81,920,000</span></td><td><span style="font-size: x-small;">6.75E-05 ms</span></td><td><span style="font-size: x-small;">6.83E-05 ms</span></td><td><span style="font-size: x-small;">1.04E-04 ms</span></td><td><span style="font-size: x-small;">3.86E-04 ms</span></td><td><span style="font-size: x-small;">8.66E-05 ms</span></td><td><span style="font-size: x-small;">6.89E-04 ms</span></td><td><span style="font-size: x-small;">2.15E-04 ms</span></td></tr>
</tbody></table>
<br />
<b>Raw data – average number of comparisons per search</b><br />
<br />
<div>
<table class="tableizer-table">
<tbody>
<tr class="tableizer-firstrow"><th><span style="font-size: x-small;">Size</span></th><th><span style="font-size: x-small;">Arrays.</span><br />
<span style="font-size: x-small;">binarySearch</span></th><th><span style="font-size: x-small;">Interpolation</span><br />
<span style="font-size: x-small;">+Seq</span></th><th><span style="font-size: x-small;">Interpolation</span></th><th><span style="font-size: x-small;">Sampling</span></th><th><span style="font-size: x-small;">Binary</span></th><th><span style="font-size: x-small;">Gallop</span></th><th><span style="font-size: x-small;">Gallop</span><br />
<span style="font-size: x-small;">+Binary</span></th></tr>
<tr><td><span style="font-size: x-small;">10,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">10.6</span></td><td><span style="font-size: x-small;">17.6</span></td><td><span style="font-size: x-small;">19.0</span></td><td><span style="font-size: x-small;">12.2</span></td><td><span style="font-size: x-small;">58.2</span></td><td><span style="font-size: x-small;">13.2</span></td></tr>
<tr><td><span style="font-size: x-small;">20,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">11.3</span></td><td><span style="font-size: x-small;">20.7</span></td><td><span style="font-size: x-small;">19.0</span></td><td><span style="font-size: x-small;">13.2</span></td><td><span style="font-size: x-small;">66.3</span></td><td><span style="font-size: x-small;">14.2</span></td></tr>
<tr><td><span style="font-size: x-small;">40,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">11.0</span></td><td><span style="font-size: x-small;">16.9</span></td><td><span style="font-size: x-small;">20.9</span></td><td><span style="font-size: x-small;">14.2</span></td><td><span style="font-size: x-small;">74.9</span></td><td><span style="font-size: x-small;">15.2</span></td></tr>
<tr><td><span style="font-size: x-small;">80,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">12.1</span></td><td><span style="font-size: x-small;">19.9</span></td><td><span style="font-size: x-small;">38.0</span></td><td><span style="font-size: x-small;">15.2</span></td><td><span style="font-size: x-small;">84.0</span></td><td><span style="font-size: x-small;">16.2</span></td></tr>
<tr><td><span style="font-size: x-small;">160,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">11.7</span></td><td><span style="font-size: x-small;">18.3</span></td><td><span style="font-size: x-small;">38.0</span></td><td><span style="font-size: x-small;">16.2</span></td><td><span style="font-size: x-small;">93.6</span></td><td><span style="font-size: x-small;">17.2</span></td></tr>
<tr><td><span style="font-size: x-small;">320,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">12.4</span></td><td><span style="font-size: x-small;">25.3</span></td><td><span style="font-size: x-small;">38.2</span></td><td><span style="font-size: x-small;">17.2</span></td><td><span style="font-size: x-small;">103.8</span></td><td><span style="font-size: x-small;">18.2</span></td></tr>
<tr><td><span style="font-size: x-small;">640,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">12.4</span></td><td><span style="font-size: x-small;">19.0</span></td><td><span style="font-size: x-small;">41.6</span></td><td><span style="font-size: x-small;">18.2</span></td><td><span style="font-size: x-small;">114.4</span></td><td><span style="font-size: x-small;">19.2</span></td></tr>
<tr><td><span style="font-size: x-small;">1,280,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">12.5</span></td><td><span style="font-size: x-small;">20.2</span></td><td><span style="font-size: x-small;">57.0</span></td><td><span style="font-size: x-small;">19.2</span></td><td><span style="font-size: x-small;">125.5</span></td><td><span style="font-size: x-small;">20.2</span></td></tr>
<tr><td><span style="font-size: x-small;">2,560,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">12.8</span></td><td><span style="font-size: x-small;">22.7</span></td><td><span style="font-size: x-small;">57.0</span></td><td><span style="font-size: x-small;">20.2</span></td><td><span style="font-size: x-small;">137.1</span></td><td><span style="font-size: x-small;">21.2</span></td></tr>
<tr><td><span style="font-size: x-small;">5,120,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">12.7</span></td><td><span style="font-size: x-small;">26.5</span></td><td><span style="font-size: x-small;">57.5</span></td><td><span style="font-size: x-small;">21.2</span></td><td><span style="font-size: x-small;">149.2</span></td><td><span style="font-size: x-small;">22.2</span></td></tr>
<tr><td><span style="font-size: x-small;">10,240,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">13.2</span></td><td><span style="font-size: x-small;">25.2</span></td><td><span style="font-size: x-small;">62.1</span></td><td><span style="font-size: x-small;">22.2</span></td><td><span style="font-size: x-small;">161.8</span></td><td><span style="font-size: x-small;">23.2</span></td></tr>
<tr><td><span style="font-size: x-small;">20,480,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">13.4</span></td><td><span style="font-size: x-small;">23.4</span></td><td><span style="font-size: x-small;">76.0</span></td><td><span style="font-size: x-small;">23.2</span></td><td><span style="font-size: x-small;">175.0</span></td><td><span style="font-size: x-small;">24.2</span></td></tr>
<tr><td><span style="font-size: x-small;">40,960,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">13.4</span></td><td><span style="font-size: x-small;">21.9</span></td><td><span style="font-size: x-small;">76.1</span></td><td><span style="font-size: x-small;">24.2</span></td><td><span style="font-size: x-small;">188.6</span></td><td><span style="font-size: x-small;">25.2</span></td></tr>
<tr><td><span style="font-size: x-small;">81,920,000</span></td><td><span style="font-size: x-small;">?</span></td><td><span style="font-size: x-small;">14.0</span></td><td><span style="font-size: x-small;">19.7</span></td><td><span style="font-size: x-small;">77.0</span></td><td><span style="font-size: x-small;">25.2</span></td><td><span style="font-size: x-small;">202.7</span></td><td><span style="font-size: x-small;">26.2</span></td></tr>
</tbody></table>
</div>
</div>
<br />
<b>Source code</b>
<br />
<br />
<a href="https://github.com/adam-ho/misc/tree/master/searchPerformance/src/main/java/com/search" rel="nofollow" target="_blank">The full Java source code for the search algorithms can be found here</a>. Keep in mind, the code is not production quality; for instance, it may have too many or too few range checks in some cases!</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com1tag:blogger.com,1999:blog-8858921836121088131.post-54822949754599710002014-05-26T17:59:00.001-07:002014-05-26T18:00:02.115-07:00Java 8 parallel sort - internals<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjy2_3uTELmJYG0vStCW50oPqhsY3DPmq_Z3T9L8JLkgrZFs1dNcgFiJz0kp0dek2PrBmAepP2d0yFl3C-R1RGV9uz2QSC-S2-qdfcbrjzAP61tERMCTEr_u8HcgBrlVUJW1wdq0kko9sc/s1600/pallelel.png" height="93" width="200" /></div>
Java 8 is a quite big update to the platform and luckily they focused on multicore systems too. The new <span style="font-family: Courier New, Courier, monospace;">Arrays.parallelSort()</span> implementation is quite interesting: it is using quicksort, merge sort, and Timsort to achieve the best performance, depending on the underlying data type and size.<br />
<br />
<b>Arrays.parallelSort() – not always parallel</b><br />
<br />
The first thing that needs to be noted is that it’s not always a parallel sort. If the array size is small enough (<= 8192 elements) or the number of reported logical CPUs is 1, it always falls back to a dual pivot quick sort. Note, that the “logical CPU” here refers to hyper-threading, so an average i7 dual core CPU will report 4 logical CPUs to the Java environment.<br />
<br />
To override the number of reported CPUs for the sort algorithm, we simply need to pass in any value to the following system property when starting the JVM:<br />
<span style="font-family: 'Courier New', Courier, monospace;">java.util.concurrent.ForkJoinPool.common.parallelism</span><br />
<br />
<b>Sorting the chunks and chunk sizes</b><br />
<br />
The algorithm (<span style="font-family: Courier New, Courier, monospace;">Arrays.java</span>) will decide on the chunk size before submitting to sort the array using the helper (<span style="font-family: Courier New, Courier, monospace;">ArraysParallelSortHelpers.java</span>). The helper also receives a working buffer as this parallel sort implementation will require the same amount of memory as the original array. This property itself may make it unsuitable for certain cases, like almost sorted arrays or extremely large arrays.<br />
<br />
The chunk size that will be distributed among CPUs will be at least 8192 items, or Array size / (4 * reported CPUs) if the latter is larger.<br />
So, an array of 500.000 items with a standard 2 core, 4 virtual cores machine would have:<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">500.000 / (4 * 4) = 31.250 items / chunk</span><br />
<br />
The original array is split into these 4 logical blocks, which are sorted then merged together. With the 4 logical processing units, each quarter is split into another quarter (16th), then sorted and merged back to the original array.<br />
<br />
The sorting itself is always the dual pivot quick sort for raw types (<span style="font-family: Courier New, Courier, monospace;">int, byte, short…</span>) and TimSort for anything else.<br />
<br />
Interestingly, the helper class has native implementation for all raw types for performance reasons, instead of using generics.<br />
<br />
<b>But why does the 8192 chunk size work so well?</b><br />
<br />
From the <span style="font-family: Courier New, Courier, monospace;">Arrays.java</span> documentation it seems like a totally empirical value: “<i><span style="font-family: Times, Times New Roman, serif;">Using smaller sizes typically results in memory contention across tasks that makes parallel speedups unlikely.</span></i>” So contention is definitely a reason, but if we look at the current CPU architectures, there is something interesting about L1 cache sizes too:<br />
<br />
Haswell (current) L1 64kb (32kb data, 32kb instructions) per core.<br />
Skylake (future) L1 128kb (64kb+64kb) per core.<br />
<br />
Fitting everything in L1 cache would be ideal, but 32kb for data on the typical 64 bit systems would mean only 4096 pointers (items in the array):<br />
<span style="font-family: 'Courier New', Courier, monospace;">32768 * 8 / 64 = 4096</span><br />
<br />
So why is the 8192 working that well? The answer is more around Java runtime: the<br />
<span style="font-family: Courier New, Courier, monospace;">-XX:+UseCompressedOops </span><br />
<br />
flag is on by default enabled for systems less than 32gb of memory on 64bit JVM since Java SE 6u23. This flag “compresses” pointers and stores the 64 bit pointers as 32 bit, sparing significant memory.<br />
<br />
<b>How about Skylake and later?</b><br />
<br />
The chunk size works very well for current CPU architectures, but as L1 cache size will grow fairly soon, this may not be the perfect size. Unfortunately the 8129 chunk size is hard wired and cannot be overridden, unlike the number of virtual CPUs. This would mean that on a newer generation CPUs the allocation of new working buffer, distributing workload across CPUs then re-merging the result may not be more efficient than single threaded sorting for not too large arrays.<br />
<br />
<b>Timsort</b><br />
<br />
Timsort is part of Python since version 2.3 and Java since 7. The idea behind this algorithm is that with real world data the array is almost never random – it contains smaller and larger sorted sub-arrays, called “runs”. The algorithm keeps identifying these runs and merges them together – hence it needs a working buffer for the merge operations.<br />
<br />
Although it’s not possible to beat the <span style="font-family: Courier New, Courier, monospace;">O(n*log(n))</span> lower bound in sorting, Timsort can be very efficient when the array is already sorted or has large "runs" that are sorted. It’s lower bound is <span style="font-family: Courier New, Courier, monospace;">O(n)</span>, unlike quicksort, which is<span style="font-family: Courier New, Courier, monospace;"> O(n*log(n))</span>; upper bound is still <span style="font-family: Courier New, Courier, monospace;">O(n*log(n))</span>, unlike quicksort, which is <span style="font-family: Courier New, Courier, monospace;">O(n^2)</span>.<br />
<br />
Apart from using working buffer, the other issue with Timsort is that the implementation is quite complex: finding the runs, merging them, then finding the correct place for the next run using galloping search requires much more complex implementation than quicksort. The Java implementation is almost 1000 lines, while a quicksort implementation can fit on a screen (around 100 lines).<br />
<br />
<i>For source code reference, have a look at <a href="http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/Arrays.java" rel="nofollow" target="_blank">Arrays.java</a> and <a href="http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/ArraysParallelSortHelpers.java" rel="nofollow" target="_blank">ArraysParallelSortHelpers.java</a></i></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com7tag:blogger.com,1999:blog-8858921836121088131.post-54002545321480343452014-05-04T02:38:00.005-07:002022-03-25T17:28:31.343-07:00Git cheat sheet - the most useful commands<div dir="ltr" style="text-align: left;" trbidi="on">
After a little practise, Git can be fully utilised from the command line without any GUI. Adding, committing, pulling, and pushing is part of the daily work; however, there are other commands that are useful but not that frequently used. Here is my collection of those:<br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Create and checkout new branch based on the current branch</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git checkout -b <newBranch></span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Undo all local pending changes</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git reset --hard HEAD</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace;"># Erase the last 3 commits</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git reset --hard HEAD~3</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Undo a single file change</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git checkout -- <fileName></span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Push new branch to server</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git push origin <branchName></span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Delete remote branch</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git push origin --delete <branchName></span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># List all local and remote branches</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git branch -a</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Remove remotely deleted branches - will not delete local branches, only copies of remotes</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git remote prune origin</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Pull remote master branch and merge it into current branch</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git pull origin master</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Undo last commit but keep changes unstaged</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git reset HEAD~1</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Rename current branch</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git branch -m <newname></span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Save changes temporarily - branch independent</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git stash</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># List temporary saves</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git stash list</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Apply temporarily saved changes</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git stash apply</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Delete temporary saves</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git stash drop</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Delete untracked files and directories (-n to just test)</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git clean -fd</span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;"># Resolve conflict by accepting their changes</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git checkout --theirs <file></span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace;"># Interactive stage / unstage</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git add -i</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<b><span style="font-family: Courier New, Courier, monospace;"># Compare branch to another branch</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git diff <otherBranch></span><br />
<br />
<b><span style="font-family: Courier New, Courier, monospace;">#display all changes with additions/deletions between two branches</span></b><br />
<span style="font-family: Courier New, Courier, monospace;">git diff --stat master..<otherBranch></span></div><div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: Courier New, Courier, monospace;"><br /></span></div><div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: Courier New, Courier, monospace;"><b style="font-family: Times;"><span style="font-family: Courier New, Courier, monospace;">#display the active branches</span></b><br style="font-family: Times;" /><span style="font-family: Courier New, Courier, monospace;">git branch -va --sort=committerdate</span></span></div><div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: Courier New, Courier, monospace;"><span style="font-family: Courier New, Courier, monospace;"><br /></span></span></div><div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: Courier New, Courier, monospace;"><span style="font-family: Courier New, Courier, monospace;"><b style="font-family: Times;"><span style="font-family: Courier New, Courier, monospace;">#display the active branches - fancy colours and author info</span></b></span></span></div><div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-family: Courier New, Courier, monospace;"><span style="font-family: Courier New, Courier, monospace;"><div dir="ltr" trbidi="on">git branch -r --sort=committerdate --format='%(HEAD) %(color:yellow)%(refname:short)%(color:reset) - %(color:red)%(objectname:short)%(color:reset) - %(contents:subject) - %(authorname) (%(color:green)%(committerdate:relative)%(color:reset))'</div><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on"><b># Working with github forks</b></div><div dir="ltr" trbidi="on"><b># If you want to keep working on your work, you need to hard-reset it to the owner's changes</b></div><div dir="ltr" trbidi="on"><div dir="ltr" trbidi="on"><br /></div><div dir="ltr" trbidi="on">git remote add upstream https://github.com/owner/repo</div><div dir="ltr" trbidi="on">git fetch upstream</div><div dir="ltr" trbidi="on">git checkout master</div><div dir="ltr" trbidi="on">git stash <b># In case you have local changes, you *must* stash them</b></div><div dir="ltr" trbidi="on">git reset --hard upstream/master </div><div dir="ltr" trbidi="on">git push origin master --force</div></div><div>git stash apply <b># If you had local changes before update</b></div></span></span></div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-25463865478656636872013-10-19T16:46:00.001-07:002013-10-19T16:46:25.326-07:00Creating a simple TCP server in Java and using it from Python – to use the Stanford POS tagger<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxbkWCSGpUDzo7mzD_hj5YTW_srfTmPAxk123gISptiUjWYxDI1LwkfI3620BgfMonPCf_Q3aPx4YmIp8lKzGiU1PBe57kEubGKC9zfE26Qh1TY8c8SbdBDZQgySlovRx-8IPHz2n-8vk/s200/java-python-tcp.png" width="200" /></div>
As the advanced network protocols flood the market we forgot how easy it is to create a simple server and a client if we want to integrate two different platforms – Java and Python in this case. The standard TCP socket library is there on practically every platform so connecting to a simple TCP server or actually creating one is just a couple of lines even in C. If we need to, we can build more complex protocols and use more complex document formats, but it's important to remember that we don't need it all the time.<br />
<br />
<b><br /></b>
<b>POS tagging</b><br />
<br />
For one of my projects, I needed to use the <a href="http://nlp.stanford.edu/software/tagger.shtml" rel="nofollow" target="_blank">Stanford POS tagger</a> to parse a large text corpus. Even though there are <a href="https://code.google.com/p/nltk/" rel="nofollow" target="_blank">Python POS taggers</a>, my favourite one is by far the Java based Stanford implementation. Usually I use it directly from Java, but in this case the input file was a bit tricky to parse and Python did it very well so I just wanted to do the POS tagging in Java and everything else in Python.<br />
<br />
My first thought was to create a file based interaction between them but that wasn't as responsive as I wanted it to be – this type of batch processing wasn't too appealing. Then for a moment I was considering using more advanced techniques like <a href="http://thrift.apache.org/" rel="nofollow" target="_blank">Apache Thrift</a> or <a href="http://code.google.com/p/protobuf/" rel="nofollow" target="_blank">Google Protobuf</a> but why would I need them if I just need to send a sentence over the write and receive the POS tagger version of it?<br />
<br />
<b>Creating the Java server</b><br />
<br />
The server part seemed to be the trickiest one as once in a while I received exception I wasn't really foreseeing. As I didn't really care about those so putting the server routine in a simple try catch solved all the issues.<br />
This is the simplified version of the server code:<br />
<br />
<pre class="brush:csharp">// Only listen on localhost, no remote connection allowed.
ServerSocket serverSocket =
new ServerSocket(10007, 0, InetAddress.getByName("localhost"));
while(true) // We never really terminate it. CTRL+C is enough.
{
System.out.println ("Waiting for connection.....");
Socket clientSocket = serverSocket.accept();
System.out.println ("Waiting for input.....");
// Create IO streams.
PrintWriter out = new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader in =
new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String inputLine;
try
{
while ((inputLine = in.readLine()) != null)
{
String tagged = tagger.tagString(inputLine); // POS tag the input.
out.println(tagged); // Send back the tagged output.
}
out.close(); // The client decided to close the connection, cleanup.
in.close();
clientSocket.close();
}
catch(Exception ex)
{
}
finally
{
System.out.println("Client has disconnected.");
}
}
// serverSocket.close(); // We never reach this.
</pre>
<br />
There are couple of interesting things I've found. One is that the Java server gets upset and throws an exception if the client decides to disappear instead of cleanly closing the socket, so a try-catch-finally was required. The other thing is that both reading and writing has to be buffered, otherwise the performance would have been really poor. The PrintWriter is buffered, so it will write the whole line to the wire in once instead of byte-by-byte.<br />
<br />
In my case the input and output was standard English text but for unicode some kind of encoding might be required.<br />
<br />
<b>Connecting from Python</b><br />
<br />
The Python client was unexpectedly easy, not counting the import it was three lines all together:<br />
<br />
<pre>import telnetlib
HOST = "localhost"
PORT = 10007
tn = telnetlib.Telnet(HOST, PORT)
tn.write(title)
response = tn.read_until("\n")
</pre>
<br />
<b>Performance and issues</b><br />
<br />
The text corpus I was parsing was quite big (10+Gb) so I quickly realised it won't be extremely fast to parse all the text. I was using an old Linux server (512Mb, Core2Duo) to parse the data overnight. The total process was around 20 hours with an average of ~100 sentences tagged a second.<br />
<br />
The only issue I was facing was that the Stanford POS tagger once in a while ran out of heap memory so I had to increase the initial heap size, but 500mb seemed to be enough (-mx500m).<br />
Otherwise the process was surprisingly stable and performing well even on that really old machine. The POS tagger did not leak memory at all so I wouldn't hesitate running the same setup again next time.</div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0tag:blogger.com,1999:blog-8858921836121088131.post-2800602398784991642013-09-04T20:32:00.000-07:002013-09-04T20:32:20.843-07:00Drawing circle and calculating sinus function without trigonometry, power, or root<div dir="ltr" style="text-align: left;" trbidi="on">
The formula of a circle is quite straightforward (<span style="font-family: Courier New, Courier, monospace;">r=sqrt(x^2+y^2)</span>) but it's not trivial how to draw a circle or calculate the trigonometric functions without advanced math. However, an interesting finding from 1972 makes it really easy.<br />
<br />
Minsky discovered (by a mistake) that the following loop will draw an almost perfect circle on the screen:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">loop:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> x = x - epsilon * y</span><br />
<span style="font-family: Courier New, Courier, monospace;"> y = y + epsilon * x <span style="color: #cccccc;"># NOTE: the x is the new x value here</span></span><br />
<br />
Turns out that epsilon in the equation is practically the rotation angle in radians, so the above loop will gradually rotate the x and y coordinates in a circle.<br />
<br />
<b>Calculating sinus</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge3HwiM0LQtj19tarAuP38UhqfxuoY7Ok-_O24TXdeX3gUhuQIqmN72h3_jKGEdnx2o3IYbhChyOPGeUyDUif15GxKVAuwNEQY0OA1UCi82_XN3N5zdZdKS_lHQRhtXxl7-o6EunQ_MB8/s1600/circle-sinus.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="189" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge3HwiM0LQtj19tarAuP38UhqfxuoY7Ok-_O24TXdeX3gUhuQIqmN72h3_jKGEdnx2o3IYbhChyOPGeUyDUif15GxKVAuwNEQY0OA1UCi82_XN3N5zdZdKS_lHQRhtXxl7-o6EunQ_MB8/s200/circle-sinus.png" width="200" /></a></div>
If we can draw this circle, we can easily estimate the sinus values: for the current angle, which is basically the sum of epsilons so far, we have a height (y), which just needs to be normalized to the 0-1 range to get the actual sinus for the angle.<br />
<br />
The smaller the steps (epsilon) are, the more accurate the formula will be. However, because it's not a perfect circle, it can never be a very accurate estimation. If epsilon is large, the algorithm will draw a visible ellipsis instead of a circle (slightly tilted left).<br />
<br />
<b><br /></b>
<br />
<b>The code in JavaScript</b><br />
<br />
I've used a simple canvas object to draw on, so the core logic of the drawing looks like this:<br />
<br />
<pre class="brush:csharp">for(var i = 0; i < steps; i++)
{
x -= y/epsilon;
y += x/epsilon;
// Draws circle.
context.fillRect(middleX+x, middleY+y, 1, 1);
var deg = i / 2 / epsilon / Math.PI * 360; // Angle in degrees.
var rad = i / epsilon; // Angle in radians.
var sinVal = y / 100; // Normalising to 0-1
// Draws sinus wave.
context.fillRect(sinX+deg, sinY+y, 1, 1);
if(i % parseInt(steps / sinMarkerCount) == 0)
{
var text = "Sin(" + parseInt(deg) + ")=" + sinVal.toFixed(2);
// Draws couple sinus values on sinus wave.
context.fillText(text, sinX+deg, sinY+y);
}
}
</pre>
<br />
<br />
And the result is this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqF0PV0D2BGyKwfEkyE9jQmAaoDig1L14sx8ea2fp3Kfp_KDdC9kLU1nhqnoTjIMlOo_P6VOg_oruGkcSi6qaMRPIGiiroLsCQm9s_e679XFYMWYCQKoRp9ilM2qXPvMVI6F-QVYK_ur4/s1600/circle-simple.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="113" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqF0PV0D2BGyKwfEkyE9jQmAaoDig1L14sx8ea2fp3Kfp_KDdC9kLU1nhqnoTjIMlOo_P6VOg_oruGkcSi6qaMRPIGiiroLsCQm9s_e679XFYMWYCQKoRp9ilM2qXPvMVI6F-QVYK_ur4/s320/circle-simple.png" width="320" /></a></div>
<br />
<br />
To grab the <a href="https://github.com/adam-ho/misc/blob/master/drawCircle/circle.html" rel="nofollow" target="_blank">full source, go here</a>. Or have a look at the <a href="http://htmlpreview.github.io/?https://github.com/adam-ho/misc/blob/master/drawCircle/circle.html" rel="nofollow" target="_blank">live demo here</a>. </div>
Adamhttp://www.blogger.com/profile/09720738376586525885noreply@blogger.com0