benchmarking – How to use clock() in C++

benchmarking – How to use clock() in C++

#include <iostream>
#include <cstdio>
#include <ctime>

int main() {
    std::clock_t start;
    double duration;

    start = std::clock();

    /* Your algorithm here */

    duration = ( std::clock() - start ) / (double) CLOCKS_PER_SEC;

    std::cout<<printf: << duration <<n;
}

An alternative solution, which is portable and with higher precision, available since C++11, is to use std::chrono.

Here is an example:

#include <iostream>
#include <chrono>
typedef std::chrono::high_resolution_clock Clock;

int main()
{
    auto t1 = Clock::now();
    auto t2 = Clock::now();
    std::cout << Delta t2-t1:  
              << std::chrono::duration_cast<std::chrono::nanoseconds>(t2 - t1).count()
              <<  nanoseconds << std::endl;
}

Running this on ideone.com gave me:

Delta t2-t1: 282 nanoseconds

benchmarking – How to use clock() in C++

clock() returns the number of clock ticks since your program started. There is a related constant, CLOCKS_PER_SEC, which tells you how many clock ticks occur in one second. Thus, you can test any operation like this:

clock_t startTime = clock();
doSomeOperation();
clock_t endTime = clock();
clock_t clockTicksTaken = endTime - startTime;
double timeInSeconds = clockTicksTaken / (double) CLOCKS_PER_SEC;

cloning – Fastest Way to do Shallow Copy in C#

cloning – Fastest Way to do Shallow Copy in C#

This is a complex subject with lots of possible solutions and many pros and cons to each. There is a wonderful article here that outlines several different ways of making a copy in C#. To summarize:

  1. Clone Manually
    Tedious, but high level of control.
  2. Clone with MemberwiseClone
    Only creates a shallow copy, i.e. for reference-type fields the original object and its clone refer to the same object.
  3. Clone with Reflection
    Shallow copy by default, can be re-written to do deep copy. Advantage: automated. Disadvantage: reflection is slow.
  4. Clone with Serialization
    Easy, automated. Give up some control and serialization is slowest of all.
  5. Clone with IL, Clone with Extension Methods
    More advanced solutions, not as common.

Id like to start with a few quotes:

In fact, MemberwiseClone is usually much better than others, especially for complex type.

and

Im confused. MemberwiseClone() should annihilate the performance of anything else for a shallow copy. […]

Theoretically the best implementation of a shallow copy is a C++ copy constructor: it knows the size compile-time, and then does a memberwise clone of all fields. The next best thing is using memcpy or something similar, which is basically how MemberwiseClone should work. This means, in theory it should obliterate all other possibilities in terms of performance. Right?

… but apparently it isnt blazing fast and it doesnt obliterate all the other solutions. At the bottom Ive actually posted a solution thats over 2x faster. So: Wrong.

Testing the internals of MemberwiseClone

Lets start with a little test using a simple blittable type to check the underlying assumptions here about performance:

[StructLayout(LayoutKind.Sequential)]
public class ShallowCloneTest
{
    public int Foo;
    public long Bar;

    public ShallowCloneTest Clone()
    {
        return (ShallowCloneTest)base.MemberwiseClone();
    }
}

The test is devised in such a way that we can check the performance of MemberwiseClone agaist raw memcpy, which is possible because this is a blittable type.

To test by yourself, compile with unsafe code, disable the JIT suppression, compile release mode and test away. Ive also put the timings after every line thats relevant.

Implementation 1:

ShallowCloneTest t1 = new ShallowCloneTest() { Bar = 1, Foo = 2 };
Stopwatch sw = Stopwatch.StartNew();
int total = 0;
for (int i = 0; i < 10000000; ++i)
{
    var cloned = t1.Clone();                                    // 0.40s
    total += cloned.Foo;
}

Console.WriteLine(Took {0:0.00}s, sw.Elapsed.TotalSeconds);

Basically I ran these tests a number of times, checked the assembly output to ensure that the thing wasnt optimized away, etc. The end result is that I know approximately how much seconds this one line of code costs, which is 0.40s on my PC. This is our baseline using MemberwiseClone.

Implementation 2:

sw = Stopwatch.StartNew();

total = 0;
uint bytes = (uint)Marshal.SizeOf(t1.GetType());
GCHandle handle1 = GCHandle.Alloc(t1, GCHandleType.Pinned);
IntPtr ptr1 = handle1.AddrOfPinnedObject();

for (int i = 0; i < 10000000; ++i)
{
    ShallowCloneTest t2 = new ShallowCloneTest();               // 0.03s
    GCHandle handle2 = GCHandle.Alloc(t2, GCHandleType.Pinned); // 0.75s (+ Free call)
    IntPtr ptr2 = handle2.AddrOfPinnedObject();                 // 0.06s
    memcpy(ptr2, ptr1, new UIntPtr(bytes));                     // 0.17s
    handle2.Free();

    total += t2.Foo;
}

handle1.Free();
Console.WriteLine(Took {0:0.00}s, sw.Elapsed.TotalSeconds);

If you look closely at these numbers, youll notice a few things:

  • Creating an object and copying it will take roughly 0.20s. Under normal circumstances this is the fastest possible code you can have.
  • However, to do that, you need to pin and unpin the object. That will take you 0.81 seconds.

So why is all of this so slow?

My explanation is that it has to do with the GC. Basically the implementations cannot rely on the fact that memory will stay the same before and after a full GC (The address of the memory can be changed during a GC, which can happen at any moment, including during your shallow copy). This means you only have 2 possible options:

  1. Pinning the data and doing a copy. Note that GCHandle.Alloc is just one of the ways to do this, its well known that things like C++/CLI will give you better performance.
  2. Enumerating the fields. This will ensure that between GC collects you dont need to do anything fancy, and during GC collects you can use the GC ability to modify the addresses on the stack of moved objects.

MemberwiseClone will use method 1, which means youll get a performance hit because of the pinning procedure.

A (much) faster implementation

In all cases our unmanaged code cannot make assumptions about the size of the types and it has to pin data. Making assumptions about size enables the compiler to do better optimizations, like loop unrolling, register allocation, etc. (just like a C++ copy ctor is faster than memcpy). Not having to pin data means we dont get an extra performance hit. Since .NET JITs to assembler, in theory this means that we should be able to make a faster implementation using simple IL emitting, and allowing the compiler to optimize it.

So to summarize on why this can be faster than the native implementation?

  1. It doesnt require the object to be pinned; objects that are moving around are handled by the GC — and really, this is relentlessly optimized.
  2. It can make assumptions about the size of the structure to copy, and therefore allows for better register allocation, loop unrolling, etc.

What were aiming for is the performance of raw memcpy or better: 0.17s.

To do that, we basically cannot use more than just a call, create the object, and perform a bunch of copy instructions. It looks a bit like the Cloner implementation above, but some important differences (most significant: no Dictionary and no redundant CreateDelegate calls). Here goes:

public static class Cloner<T>
{
    private static Func<T, T> cloner = CreateCloner();

    private static Func<T, T> CreateCloner()
    {
        var cloneMethod = new DynamicMethod(CloneImplementation, typeof(T), new Type[] { typeof(T) }, true);
        var defaultCtor = typeof(T).GetConstructor(new Type[] { });

        var generator = cloneMethod .GetILGenerator();

        var loc1 = generator.DeclareLocal(typeof(T));

        generator.Emit(OpCodes.Newobj, defaultCtor);
        generator.Emit(OpCodes.Stloc, loc1);

        foreach (var field in typeof(T).GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
        {
            generator.Emit(OpCodes.Ldloc, loc1);
            generator.Emit(OpCodes.Ldarg_0);
            generator.Emit(OpCodes.Ldfld, field);
            generator.Emit(OpCodes.Stfld, field);
        }

        generator.Emit(OpCodes.Ldloc, loc1);
        generator.Emit(OpCodes.Ret);

        return ((Func<T, T>)cloneMethod.CreateDelegate(typeof(Func<T, T>)));
    }

    public static T Clone(T myObject)
    {
        return cloner(myObject);
    }
}

Ive tested this code with the result: 0.16s. This means its approximately 2.5x faster than MemberwiseClone.

More importantly, this speed is on-par with memcpy, which is more or less the optimal solution under normal circumstances.

Personally, I think this is the fastest solution – and the best part is: if the .NET runtime will get faster (proper support for SSE instructions etc), so will this solution.

Editorial Note:
The sample code above assumes that the default constructor is public. If it is not, the call to GetConstructor returns null. In that case, use one of the other GetConstructor signatures to obtain protected or private constructors.
See https://docs.microsoft.com/en-us/dotnet/api/system.type.getconstructor?view=netframework-4.8

cloning – Fastest Way to do Shallow Copy in C#

Im confused. MemberwiseClone() should annihilate the performance of anything else for a shallow copy. In the CLI, any type other than an RCW should be able to be shallow-copied by the following sequence:

  • Allocate memory in the nursery for the type.
  • memcpy the data from the original to the new. Since the target is in the nursery, no write barriers are required.
  • If the object has a user-defined finalizer, add it to the GC list of items pending finalization.
    • If the source object has SuppressFinalize called on it and such a flag is stored in the object header, unset it in the clone.

Can someone on the CLR internals team explain why this is not the case?

notepad++ – Remove a BOM character in a file

notepad++ – Remove a BOM character in a file

If you look in the same menu. Click Convert to UTF-8.

If

You can solve the problem using vim, where you can get easily with MinGW-w64 (If you have installed Git it comes along) or Cygwin.

So, the key is to use:

  • The option -s, which will execute a vim script with vim commands.
  • The option -b, which will open your file in binary mode, where youll see those awkward BOM bytes
  • The option -n, which is very important! This option refuses the use of swap files, so all your work runs in memory. It gives you assurance because if the file is large, the swap files can mislead the process.

That said, lets go to the code!

  1. First you create a simple file, here named script, which will hold the vim commands
    echo gg+gPggdtCZZ > script
    

    …this weird string says to vim Go to the beginning of the file, copy the first word and paste it behind the cursor, so delete everything until character C, then, save the file

    Note: If your file starts with other character than C, you have to specify it. If you have different first characters, you can follow the logic and create a bash script which will read the first character and replace it for you in the snippet above.

  2. Run the vim command:
    vim -n -b <the_file> -s script
    

notepad++ – Remove a BOM character in a file

I believe this is not to be seen as a problem. When it is a problem BOM are just 3 bytes EF BB BF . Can not we just delete this? Or change to something and then closing the file again?

Anyway this thing below can do the trick and change BOM if present to ***. Run as

x file 

where file is the name of the file.

#define _CRT_SECURE_NO_WARNINGS     
#include <stdio.h>
#include <string.h>

int main(int argc, char** argv)
{
    const unsigned char BOM[3] = { xEF, xBB, xBF };
    char file_name[64] = { target.csv };
    if (argc > 1) strcpy(file_name, argv[1]);
    FILE* one = fopen(file_name, r+b);
    if (!one) return -1;
    unsigned char buffer[64];
    int n = fread(buffer, 1, 3, one);
    if (n != 3)return -2;
    if (memcmp(buffer, BOM, 3) != 0)
    {   printf(file %s has no BOMn, file_name);
        fclose(one);
        return 0;
    };
    n = fseek(one, 0, SEEK_SET);
    if (n != 0) return -3;
    buffer[0] = buffer[1] = buffer[2] = *;
    n = fwrite(buffer, 1, 3, one);
    if (n == 3)
        printf(Byte Order Mark changed to ***n);
    else
        printf(Error writing to filen);
    fclose(one);
    return 0;
}

How to import Angular Material in project?

How to import Angular Material in project?

UPDATE for Angular 9.0.1

Since this version there is no barrel file for massive exports in the root index.d.ts. The assets imports should be:

import { NgModule } from @angular/core;
import { MatCardModule } from @angular/material/card;
import { MatButtonModule} from @angular/material/button;
import { MatMenuModule } from @angular/material/menu;
import { MatToolbarModule } from @angular/material/toolbar;
import { MatIconModule } from @angular/material/icon;

import {
  MatButtonModule,
  MatMenuModule,
  MatToolbarModule,
  MatIconModule,
  MatCardModule
} from @angular/material;

@NgModule({
  imports: [
    MatButtonModule,
    MatMenuModule,
    MatToolbarModule,
    MatIconModule,
    MatCardModule
  ],
  exports: [
    MatButtonModule,
    MatMenuModule,
    MatToolbarModule,
    MatIconModule,
    MatCardModule
  ]
})
export class MaterialModule {}

source: @angular/material/index.d.ts is not a module


MaterialModule was depreciated in version 2.0.0-beta.3 and it has been removed completely in version 2.0.0-beta.11. See this CHANGELOG for more details. Please go through the breaking changes.

Breaking changes

  • Angular Material now requires Angular 4.4.3 or greater
  • MaterialModule has been removed.
  • For beta.11, weve made the decision to deprecate the md prefix
    completely and use mat moving forward.

Please go through CHANGELOG we will get more answer!

Example shown below
cmd

npm install --save @angular/material @angular/animations @angular/cdk
npm install --save angular/material2-builds angular/cdk-builds

Create file (material.module.ts) inside the app folder

import { NgModule } from @angular/core;

import {
  MatButtonModule,
  MatMenuModule,
  MatToolbarModule,
  MatIconModule,
  MatCardModule
} from @angular/material;

@NgModule({
  imports: [
    MatButtonModule,
    MatMenuModule,
    MatToolbarModule,
    MatIconModule,
    MatCardModule
  ],
  exports: [
    MatButtonModule,
    MatMenuModule,
    MatToolbarModule,
    MatIconModule,
    MatCardModule
  ]
})
export class MaterialModule {}

import on app.module.ts

import { MaterialModule } from ./material.module;

Your component html file

<div>
  <mat-toolbar color=primary>
    <span><mat-icon>mood</mat-icon></span>

    <span>Yay, Material in Angular 2!</span>

    <button mat-icon-button [mat-menu-trigger-for]=menu>
      <mat-icon>more_vert</mat-icon>
    </button>
  </mat-toolbar>
  <mat-menu x-position=before #menu=matMenu>
    <button mat-menu-item>Option 1</button>
    <button mat-menu-item>Option 2</button>
  </mat-menu>

  <mat-card>
    <button mat-button>All</button>
    <button mat-raised-button>Of</button>
    <button mat-raised-button color=primary>The</button>
    <button mat-raised-button color=accent>Buttons</button>
  </mat-card>

  <span class=done>
    <button mat-fab>
      <mat-icon>check circle</mat-icon>
    </button>
  </span>
</div>

Add global css style.css

@import https://fonts.googleapis.com/icon?family=Material+Icons;
@import [email protected]/material/prebuilt-themes/indigo-pink.css; 

Your component css

body {
  margin: 0;
  font-family: Roboto, sans-serif;
}

mat-card {
  max-width: 80%;
  margin: 2em auto;
  text-align: center;
}

mat-toolbar-row {
  justify-content: space-between;
}

.done {
  position: fixed;
  bottom: 20px;
  right: 20px;
  color: white;
}

If any one didnt get output use below instruction

instead of above interface (material.module.ts) u can directly use below code also in the app.module.ts.

import { FormsModule, ReactiveFormsModule } from @angular/forms;
import { BrowserAnimationsModule } from @angular/platform-browser/animations;
import { MdButtonModule, MdCardModule, MdMenuModule, MdToolbarModule, MdIconModule, MatAutocompleteModule, MatInputModule,MatFormFieldModule } from @angular/material;

So this case u dont want to import

import { MaterialModule } from ./material.module;

in the app.module.ts

Import all Angular Material modules in Angular 9.

Create material.module.ts file in your_project/src/app/ directory and paste this code.

import { NgModule } from @angular/core;
import { CommonModule } from @angular/common;


import { BrowserAnimationsModule } from @angular/platform-browser/animations;
import { MatCheckboxModule } from @angular/material/checkbox;
import { MatButtonModule } from @angular/material/button;
import { MatInputModule } from @angular/material/input;
import { MatAutocompleteModule } from @angular/material/autocomplete;
import { MatDatepickerModule } from @angular/material/datepicker;
import { MatFormFieldModule } from @angular/material/form-field;
import { MatRadioModule } from @angular/material/radio;
import { MatSelectModule } from @angular/material/select;
import { MatSliderModule } from @angular/material/slider;
import { MatSlideToggleModule } from @angular/material/slide-toggle;
import { MatMenuModule } from @angular/material/menu;
import { MatSidenavModule } from @angular/material/sidenav;
import { MatBadgeModule } from @angular/material/badge;
import { MatToolbarModule } from @angular/material/toolbar;
import { MatListModule } from @angular/material/list;
import { MatGridListModule } from @angular/material/grid-list;
import { MatCardModule } from @angular/material/card;
import { MatStepperModule } from @angular/material/stepper;
import { MatTabsModule } from @angular/material/tabs;
import { MatExpansionModule } from @angular/material/expansion;
import { MatButtonToggleModule } from @angular/material/button-toggle;
import { MatChipsModule } from @angular/material/chips;
import { MatIconModule } from @angular/material/icon;
import { MatProgressSpinnerModule } from @angular/material/progress-spinner;
import { MatProgressBarModule } from @angular/material/progress-bar;
import { MatDialogModule } from @angular/material/dialog;
import { MatTooltipModule } from @angular/material/tooltip;
import { MatSnackBarModule } from @angular/material/snack-bar;
import { MatTableModule } from @angular/material/table;
import { MatSortModule } from @angular/material/sort;
import { MatPaginatorModule } from @angular/material/paginator;


@NgModule( {
    imports: [
        CommonModule,
        BrowserAnimationsModule,
        MatCheckboxModule,
        MatCheckboxModule,
        MatButtonModule,
        MatInputModule,
        MatAutocompleteModule,
        MatDatepickerModule,
        MatFormFieldModule,
        MatRadioModule,
        MatSelectModule,
        MatSliderModule,
        MatSlideToggleModule,
        MatMenuModule,
        MatSidenavModule,
        MatBadgeModule,
        MatToolbarModule,
        MatListModule,
        MatGridListModule,
        MatCardModule,
        MatStepperModule,
        MatTabsModule,
        MatExpansionModule,
        MatButtonToggleModule,
        MatChipsModule,
        MatIconModule,
        MatProgressSpinnerModule,
        MatProgressBarModule,
        MatDialogModule,
        MatTooltipModule,
        MatSnackBarModule,
        MatTableModule,
        MatSortModule,
        MatPaginatorModule

    ],
    exports: [
        MatButtonModule,
        MatToolbarModule,
        MatIconModule,
        MatSidenavModule,
        MatBadgeModule,
        MatListModule,
        MatGridListModule,
        MatInputModule,
        MatFormFieldModule,
        MatSelectModule,
        MatRadioModule,
        MatDatepickerModule,
        MatChipsModule,
        MatTooltipModule,
        MatTableModule,
        MatPaginatorModule
    ],
    providers: [
        MatDatepickerModule,
    ]
} )

export class AngularMaterialModule { }

How to import Angular Material in project?

If you want to import all Material modules, create your own module i.e. material.module.ts and do something like the following:

import { NgModule } from @angular/core;
import * as MATERIAL_MODULES from @angular/material;

export function mapMaterialModules() {
  return Object.keys(MATERIAL_MODULES).filter((k) => {
    let asset = MATERIAL_MODULES[k];
    return typeof asset == function
      && asset.name.startsWith(Mat)
      && asset.name.includes(Module);
  }).map((k) => MATERIAL_MODULES[k]);
}
const modules = mapMaterialModules();

@NgModule({
    imports: modules,
    exports: modules
})
export class MaterialModule { }

Then import the module into your app.module.ts

stl – What does the push_heap function in C++ do?

stl – What does the push_heap function in C++ do?

You are using push_heap incorrectly.

After initializing your vector, you need to put it in heap order:

std::make_heap(vector1.begin(), vector1.end());

To add further elements into the heap, you need to first push each to the back of the vector, then call push_heap:

vector1.push_back(42);
std::push_heap(vector1.begin(), vector1.end());

Finally, to remove the first element in the heap, you need to call pop_heap, followed by popping the last element from the vector:

std::pop_heap(vector1.begin(), vector1.end());
vector1.pop_back();

The three-parameter heap functions let you specify a compare method to control the heap order, which you are doing correctly.

The reason for the manual push_back and pop_back calls is that the heap functions only see iterators into a container, and do not have access to the container itself. Since iterators are not sufficient to modify the contents of a container, this must be done manually by the owner of the container (you).

To avoid having to deal with any of this yourself, Id recommend using a std::priority_queue.

This function does not turn a range of values into a heap!

std::push_heap(first, last [, comp])

assumes that the range [first,last-1) is already a valid heap and pushes the value at position last-1 into the heap, moving it to the correct position to keep the heap-requirement valid. It uses either the < operator to determine the ordering of the elements or a user-specified comparator.

stl – What does the push_heap function in C++ do?

jndi – Initialcontext in a standalone Java program

jndi – Initialcontext in a standalone Java program

Here is an example adapted from the accepted answer but doing everything inline to avoid creating extra classes.

public static void main(String[] args) {
    setupInitialContext();
    //do something that looks up a datasource
}

private static void setupInitialContext() {
    try {
        NamingManager.setInitialContextFactoryBuilder(new InitialContextFactoryBuilder() {

            @Override
            public InitialContextFactory createInitialContextFactory(Hashtable<?, ?> environment) throws NamingException {
                return new InitialContextFactory() {

                    @Override
                    public Context getInitialContext(Hashtable<?, ?> environment) throws NamingException {
                        return new InitialContext(){

                            private Hashtable<String, DataSource> dataSources = new Hashtable<>();

                            @Override
                            public Object lookup(String name) throws NamingException {

                                if (dataSources.isEmpty()) { //init datasources
                                    MysqlConnectionPoolDataSource ds = new MysqlConnectionPoolDataSource();
                                    ds.setURL(jdbc:mysql://localhost:3306/mydb);
                                    ds.setUser(mydbuser);
                                    ds.setPassword(mydbpass);
                                    dataSources.put(jdbc/mydbname, ds);

                                    //add more datasources to the list as necessary
                                }

                                if (dataSources.containsKey(name)) {
                                    return dataSources.get(name);
                                }

                                throw new NamingException(Unable to find datasource: +name);
                            }
                        };
                    }

                };
            }

        });
    }
    catch (NamingException ne) {
        ne.printStackTrace();
    }
}

You could also create your own custom context.

LocalContext ctx = LocalContextFactory.createLocalContext();
ctx.addDataSource(jdbc/testdb, driverName, url, usr, pwd);

See Running Beans Locally that use Application Server Data Sources for more details.


new UPDATE

You can use the class org.springframework.mock.jndi.SimpleNamingContextBuilder of Spring. e.g.:

  • Setup:
    SimpleNamingContextBuilder builder = new SimpleNamingContextBuilder();
    builder.bind(jdbc/Oracle, ods);
    builder.activate();
    
  • Use:
    DataSource ds = InitialContext.doLookup(jdbc/Oracle);
    

jndi – Initialcontext in a standalone Java program

You can create your own Context by sub-classing javax.naming.InitialContext and implementing only a small subset of methods, typically the bind and the lookup methods.

Then you can create your data source and bind it to your initial context to a specific key. After this you are ready to go and query from any place your JNDI context in your stand-alone Java programme.

This is the code you can use to create your own context:

InitialContext initialContext = new InitialContext() {

    private Map<String, Object> table = new HashMap<>();

    public void bind(String key, Object value) {
        table.put(key, value);
    }

    public Object lookup(String key) throws NamingException {
        return table.get(key);
    }
};

// Activate the initial context
NamingManager.setInitialContextFactoryBuilder(environment -> environment1 -> initialContext);

Then you can initialise your data source, whichever you choose:

InitialContext ic = new InitialContext();

BasicDataSource bds = new BasicDataSource();
bds.setDriverClassName(com.microsoft.sqlserver.jdbc.SQLServerDriver);
bds.setUrl(url);
bds.setUsername(username);
bds.setPassword(password);

ic.bind(jndiPath, bds);

And somewhere else in your code, you can use the existing data source by retrieving it from the JNDI context:

InitialContext ic2 = new InitialContext();
DataSource ds = (DataSource) ic2.lookup(jndiPath);
assertNotNull(ds);
Connection conn = ds.getConnection();
assertNotNull(conn);
conn.close();

multithreading – I cant understand polling/select in python

multithreading – I cant understand polling/select in python

Okay, one question a time.

What are those for?

Here is a simple socket server skeleton:

s_sock = socket.socket()
s_sock.bind()
s_sock.listen()

while True:
    c_sock, c_addr = s_sock.accept()
    process_client_sock(c_sock, c_addr)

Server will loop and accept connection from a client, then call its process function to communicate with client socket. There is a problem here: process_client_sock might takes a long time, or even contains a loop(which is often the case).

def process_client_sock(c_sock, c_addr):
    while True:
        receive_or_send_data(c_sock)

In which case, the server is unable to accept any more connections.

A simple solution would be using multi-process or multi-thread, just create a new thread to deal with request, while the main loop keeps listening on new connections.

s_sock = socket.socket()
s_sock.bind()
s_sock.listen()

while True:
    c_sock, c_addr = s_sock.accept()
    thread = Thread(target=process_client_sock, args=(c_sock, c_addr))
    thread.start()

This works of course, but not well enough considering performance. Because new process/thread takes extra CPU and memory, not idle for servers might get thousands connections.

So select and poll system calls tries to solve this problem. You give select a set of file descriptors and tell it to notify you if any fd is ready to read/write/ or exception happens.

does it(select) block while watching a resource?

Yes, or no depends on the parameter you passed to it.

As select man page says, it will get struct timeval parameter

int select(int nfds, fd_set *readfds, fd_set *writefds,
       fd_set *exceptfds, struct timeval *timeout);

struct timeval {
long    tv_sec;         /* seconds */
long    tv_usec;        /* microseconds */
};

There are three cases:

  1. timeout.tv_sec == 0 and timeout.tv_usec = 0

    No-blocking, return immediately

  2. timeout == NULL

    block forever until a file descriptor is ready.

  3. timeout is normal

    wait for certain time, if still no file descriptor is available, timeout and return.

What is the purpose of polling ?

Put it into simple words: polling frees CPU for other works when waiting for IO.

This is based on the simple facts that

  1. CPU is way more faster than IO
  2. waiting for IO is a waste of time, because for the most time, CPU will be idle

Hope it helps.

If you do read or recv, youre waiting on only one connection. If you have multiple connections, you will have to create multiple processes or threads, a waste of system resource.

With select or poll or epoll, you can monitor multiple connections with only one thread, and get notified when any of them has data available, and then you call read or recv on the corresponding connection.

It may block infinitely, block for a given time, or not block at all, depending on the arguments.

multithreading – I cant understand polling/select in python

select() takes in 3 lists of sockets to check for three conditions (read, write, error), then returns (usually shorter, often empty) lists of sockets that actually are ready to be processed for those conditions.

s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.bind((Local_IP, Port1))
s1.listen(5)

s2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s2.bind((Local_IP, Port2))
s2.listen(5)

sockets_that_might_be_ready_to_read = [s1,s2]
sockets_that_might_be_ready_to_write_to = [s1,s2]
sockets_that_might_have_errors = [s1,s2]


([ready_to_read], [ready_to_write], [has_errors])  = 
       select.select([sockets_that_might_be_ready_to_read],
                     [sockets_that_might_be_ready_to_write_to], 
                     [sockets_that_might_have_errors],            timeout)


for sock in ready_to_read:
    c,a = sock.accept()
    data = sock.recv(128)
    ...
for sock in ready_to_write:
    #process writes
    ...
for sock in has_errors:
    #process errors

So if a socket has no attempted connections after waiting timeout seconds, then the list ready_to_read will be empty – at which point it doesnt matter if the accept() and recv() would block – they wont get called for the empty list….

If a socket is ready to read, then if will have data, so it wont block then, either.

Problems with Hadoop distcp from HDFS to Amazon S3

Problems with Hadoop distcp from HDFS to Amazon S3

You should use s3n instead of s3.

s3n is the native file system implementation (ie – regular files), using s3 imposes hdfs block structure on the files so you cant really read them without going through hdfs libraries.

Thus:

hadoop distcp hdfs://file/1 s3n://bucket/destination

Amazon has created a version of distcp that is optimized for transferring between hdfs and s3 which they call, appropriately, s3distcp. You may want to check that out as well. It is intended for use with Amazon EMR, but the jar is available in s3, so you might be able to use it outside of an EMR job flow.

http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html

Problems with Hadoop distcp from HDFS to Amazon S3

In the event that your files in HDFS are larger than 5GB, you will encounter errors in your distcp job that look like:

Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version=1.0 encoding=UTF-8?><Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>23472570134</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed><RequestId>5BDA6B12B9E99CE9</RequestId><HostId>vmWvS3Ynp35bpIi7IjB7mv1waJSBu5gfrqF9U2JzUYsXg0L7/sy42liEO4m0+lh8V6CqU7PU2uo=</HostId></Error> at org.jets3t.service.S3Service.putObject(S3Service.java:2267) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:122) ... 27 more Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143

To fix this, use either the s3n filesystem as @matthew-rathbone suggested, but with -Dfs.s3n.multipart.uploads.enabled=true like:

hadoop distcp -Dfs.s3n.multipart.uploads.enabled=true hdfs://file/1 s3n://bucket/destination

OR

Use the next generation s3 filesystem, s3a like:

hadoop distcp -Dfs.s3a.endpoint=apigateway.us-east-1.amazonaws.com hdfs://file/1 s3a://bucket/destination

Options and documentation for these live here: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html

google sheets – Script to generate random numbers in multiple cells

google sheets – Script to generate random numbers in multiple cells

Here is a script that will fill each the selected cells with a random number from 1-9. It can be activated from the menu it creates called Fill random. You should be able to modify it, if needed, to suit your specific requirements:

function numbers19() {
  var sheet = SpreadsheetApp.getActiveSheet();
  var range = sheet.getActiveRange();
  for (var x = 1; x <= range.getWidth(); x++) {
    for (var y = 1; y <= range.getHeight(); y++) {
      var number = Math.floor(Math.random() * 8) + 1;
      range.getCell(y, x).setValue(number);
    }
  }
};

function onOpen() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet();
  var entries = [{
    name : Numbers: 1-9,
    functionName : numbers19
  }];
  sheet.addMenu(Fill random, entries);
};

To only change a specific range, use the following value for range:

var range = sheet.getRange(c2:f2);

google sheets – Script to generate random numbers in multiple cells

When do I need to import java.awt.*? If I want to import javax.swing.* do I need to also import java.awt.*?

When do I need to import java.awt.*? If I want to import javax.swing.* do I need to also import java.awt.*?

You import awt if you use awt types, no more, no less.

I cant remember if I need to import java.awt.* to use anything in javax.swing.* or would I have imported java.awt.* for another reason? In other words, what is the purpose of importing java.awt.* ?

The purpose is to use any classes in the AWT library. Most would recommend that you import only the specific classes that youre using, no more, no less.

I was under the impression everything I needed to use JButton, JFrame, etc. was in javax.swing.*.

This will almost never be true. Most Java programs that are more than toy programs will need to use many libraries. For instance, most layout classes are held in the AWT library. Most listeners are held in the java.awt.event library.

Note that you dont have to import anything if you choose to use fully qualified type names.

e.g.,

java.awt.BorderLayout myBorderLayout = new java.awt.BorderLayout(5, 5);

These are from javax.swing.*

JFrame
JButton

These are from java.awt.*

Component
Color

You can make a window without java.awt.*

public class Frame extends JFrame{
    public static void main(String[] args){
        new Frame();
    }
    public Frame(){
        super(Frame);
        this.setVisible(true);
    }
}

If you wanted to, I prefer not to, you can use a fully qualified class name.

javax.swing.JFrame = new javax.swing.JFrame();

If you do this, you can make a program without importing anything.

When do I need to import java.awt.*? If I want to import javax.swing.* do I need to also import java.awt.*?