Best practice for Postgres setup with docker-compose

First, use alpine images if possible.

Second, use ENV variables for prepare database configuration. More info.

Third, use /docker-entrypoint-initdb.d directory for extend image. More info.

Example stack with devlopment environment.

version: '3.2'
services:
  example:
    image: "${EXAMPLE_SERVICE}:${EXAMPLE_SERVICE__VERSION}-development"
    build:
      context: .
      dockerfile: Dockerfile.development
    networks:
      - net
    ports:
      - 5000:5000
    depends_on:
      - example__migrator
    stdin_open: true
    tty: true
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
      FLASK_APP: app.py
      FLASK_DEBUG: 1
    volumes:
      - .:/home/example
  example__migrator:
    image: "${EXAMPLE_SERVICE}:${EXAMPLE_SERVICE__VERSION}-development"
    build:
      context: .
    command: sh -c './wait-for pg.example_net:5432 -- python manage.py db upgrade'
    networks:
      - net
    depends_on:
      - pg
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
  pg:
    image: postgres:10.0-alpine
    networks:
      - net
    ports:
      - 5432:5432
    volumes:
      - pg_data:/var/lib/postgresql/data/pg_data
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
      PGDATA: /var/lib/postgresql/data/pg_data
networks:
  net:
volumes:
  pg_data:


Run JMeter Test in Jenkins with Parameters

Yes, you can easily trigger your JMeter jmx suite via jenkins coupled with performance plugin. For triggering the build:

If you're using windows execute the build as windows batch command, giving the same commands that you would give if you were running your JMeter script from your command line.
To give parameters you can conveniently select the option in jenkins configuration 'This project is parameterised' and pass the parameters ensuring the same reference is used in your JMeter jmx file.
You can use the __P() function to make ramup, hostname or number of threads configurable and refer them using J command.


String to specify to KUBE_SERVICE_ADDRESSES (API Server configuration)

According to this Q&A and this code the --portal_net was renamed to --service-cluster-ip-range since version 0.18.

# --service-cluster-ip-range=<nil>: A CIDR notation IP range from which to assign service cluster IPs.
# This must not overlap with any IP ranges assigned to nodes for pods.
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=${SERVICE_CLUSTER_IP_RANGE}"


How to keep many servers (5000+) up-to-date?

I don't think the proposed plan here makes much sense.  Tools like Ansible are free and designed to do this with good, central management and logging.

Put code in BitBucket.
Make a Jenkins server (open source / free / low on resources
/ well documented and easy to use).
Make BitBucket call Jenkins with a hook on change.

Or, if you feel like it, Jenkins can poll bit-bucket, but this is less efficient.

Have Jenkins call ansible (also open source / free / low on resources
/ well documented and easy to use) in its pipeline.
Ansible can target all 5000 servers with a dynamic inventory based on querying a database/file/API/whatever you have.
This way, bit-bucket has almost no load, ansible does everything from one location, runs are recorded in Jenkins with info on each host.

You won't have any API / rate limits here and you'll easily be able to target more hosts, with no effort on your end, with the dynamic inventory.

This is industry standard tooling.  You can also use Ansible to keep the servers up to date patch wise and all kinds of other stuff while you're at it, so it's a re-usable solution.


Which organization types are targeted by DevOps transformation?

DevOps applies to Organization Type 2, the Business to Business entity as much if not more than the first Organization type. There are two possible scenarios here:

Scenario 1

In the first they are selling a product or service to businesses. In terms of DevOps, this makes them virtually indistinguishable from Organization Type 1. Though their customer is a business/group of individuals, they are providing a product or service to a customer. This customer just happens to be a business instead of a private individual, this is is irrelevant to execution of DevOps. Contracted with specific software goals, clearly defined requirement or otherwise, the best value and customer service will be provided by a DevOps model for reasons that do not need to be restated here.

Scenario 2

This scenario happens to be a bit more exceptional as in this case, Organization Type 2 is providing consulting services to a business. Presumably and usually, a business hires a consultant for business assistance within the consultant's domain of expertise. Businesses have technical support for assistance with specific technologies and can contract en engineer for migrations, but when issues such as risk management, overall architectural design, process, flow and high-level questions arise, this is when a consultant is hired for their business advice.

This is why it is important to remember that DevOps is not a role. I would argue that DevOps represents a business or organizational management processes - or even a philosophy. It is more or less a specific way of implementing an agile methodology that has been extended well beyond just development to encompass many disciplines and departments. As such, DevOps is a package deal - in order to be successful at DevOps, you have to be committed to a specific business philosophy of organizational management.

In fact, in order to illustrate this point, In "The Phoenix Project", Board member Erik Reid repeatedly drags the Book's main character, Bill Palmer, to a parts manufacturing plant in order to illustrate and teach Bill things about technology and software development by drawing business lessons from the manufacturing sector.

This is authors Gene Kim, Kevin Behr and George Spafford's way of driving home the point that DevOps is more than just way of doing things, or specific technologies, but is a philosophy of business simply known by various nom de plumes.

So, in order to provide the best value to the company to which a consultant is consulting, it is incumbent upon the consultant to demonstrate and explain the value of DevOps and assist a company in successfully executing on this business philosophy and guide the company to specific technologies and ways of doing that. In short, it is the consulting companies job to act as an "Erik Reid" to whatever business has contracted their consulting services.

In short, DevOps should be very important to Organization Type 2 because it is their job as a consultant to explain to their customer why it is important - they should always be selling DevOps consulting. Consultants working with a company who refuses to consider DevOps should consider politely suggesting that perhaps they aren't the right fit. their customer can hire a generic code monkey from anywhere if they want to continue to (unsuccessfully) continue to do projects the way they have always done them.


How to define failure situation on jenkins

By default Jenkins assumes whether the build is SUCCESS or not from build process exit code.

Which means that build exit with code 0 considered as SUCCESS, rest all are considered as FAILURE.

If your build step/process exit with code 0, even during the failure case then Jenkins will report the build is SUCCESS.

So I my suggestion is to run the all build steps manually and try to get the exit code on each step (for both PASS and FAIL case).


How to cache node_modules folder when using Jenkins Multibranch Pipeline

It is possible to do this (not with any built-in steps; you essentially have to either use a global cache or write your own caching tool), but I do not recommend it.  When I've tried this in the past, I often ran into race conditions.  For example, one build would write to the cache while another build simultaneously reads from the cache, or two builds would write to the cache simultaneously, etc.  I got a lot of builds with corrupted modules as a result.

Caches do not have these kinds of race conditions when limited to a single job (or in this case, a single branch for a multibranch job) because Jenkins can be configured not to run multiple builds of one job simultaneously.  If you take extra steps to ensure that all of the jobs which might access the cache can't run simultaneously, then you could probably get away with a shared cache.  But at that point, you're likely to lose the added speed benefit of sharing the cache anyway, since you've massively reduced Jenkins' capacity to run jobs in parallel.

The other scenario where you could get away with a shared cache is if you write to the cache extremely infrequently - and I really do mean extremely.  If the cache is rarely written to, then there simply aren't many opportunities for these kinds of race conditions.


Are package facts getting invalidated automatically when I install remove packages?

Package facts (as any other facts like devices, disks and their partitions, IP addresses...) don't refresh automagically.
In your case, you can use a handler and notify it from any task that can possibly invalidate the cache. Here is a quickNdirty example:
- name: Pseudo code demo for package facts refresh
  hosts: localhost

  tasks:

    - name: Gather the package facts unconditionally at play start
      package_facts:

    - name: Make sure package X is installed
      package:
        name: package_X
      notify: refresh_package_facts

    # By default, handlers run at the end of the play.
    # If you want to force them to run at a certain point,
    # you need the following meta task.
    # Note => this will run !!ALL!! notified handlers
    - name: Run handlers if notified
      meta: flush_handlers

    - name: This task will run with an up-to-date cache
      debug:
        msg: "{{ ansible_facts.packages | length }} packages are installed"

  handlers:

    - name: Refresh the package facts
      package_facts:
      listen: refresh_package_facts


AWS Security group rule for certbot renewal

You just can not because from Let's Encrypt FAQ:

  What IP addresses does Let’s Encrypt use to validate my web server?

  We don’t publish a list of IP addresses we use to validate, because
  they may change at any time. In the future we may validate from
  multiple IP addresses at once.

You will need a reverse proxy (usually a WAF), routing /.well-know/acme to a machine and answering the challenges (from anywhere) and doing the IP filtering before forwarding to your application all other requests.


What's difference between on-premise and SaaS?

SaaS (Software as a Service), often referred to "on-demand Software", is a model where you're provided with access to the software in the cloud. There's no installation, setup or running of the setup, as the service provider will do this for you. You just have to pay the service provider. Example: Office 365
On-Premise is the "traditional" model where you have to install, setup and run the software at a computer at your physical location. It takes time, personnel and possible equipment. It may also have additional cost to hardware and software. Example: Office 2016

Office 365 is SaaS which provides an online version of the MS Office suite. While Office 2016 is On-Premise Software that will require installation and running.

In addition

PaaS (Platform as a Service) provides a platform which typical includes operation system, databases, web server, containers for executing programming languages. Examples: Windows Azure, Azure SQL, Heroku
IaaS (Infrastructure as a Service), provides as name suggesting infrastructure, like virtual machines, storage, network, network service (firewalls). Examples: Windows Azure, Google Compute Engine


Can be DevOps culture mapped to a specific type of organization as described by Laloux?

This question perfectly illustrates the different approaches in adopting DevOps by organizations. While both approaches target the 3rd circle on the culture model - the traditional corporations - there is a difference in aim and results.

Adopt DevOps best practices and tools for increased effectiveness.
Adopt DevOps culture and values for organizational transformation.

Established corporations often aim at (1) and either omit or fail to adopt (2). They will still gain increased effectiveness and improved organizational key performance indicators, but they will fundamentally stay at the orange level of organizational development, failing to undergo the transformation. Such transformation always requires an existential threat to the organization and those are rarely recognized and acted upon until there is too late for the organization to transform.

New organizations and organizations with new leadership have an easy way to focus on (2) and so they either are raised up at the new developmental stage or manage to successfully seed business units operating with new culture and values that either subsume the entire organization eventually or the old business units slowly decay and whither while the new ones take over a large bulk of the business. It is almost like a parent passing his family business to a child who has been raised with different values and more progressive ideas.

Either way, you can say that DevOps as it is understood today, operates between the orange and green layer and has the potential to enable the transformation from Orange to Green under certain conditions.

The core ideas behind DevOps are a different matter. They are in a way a method for an organization to pass from any developmental stage to the next, but as DevOps is being codified through specific cultural recommendations, concrete values, and actual practices and tools, the instantiation of those core ideas in form of DevOps is squarely aimed at the border between Orange and Green.


Accessing remote servers through a VPN tunnel from within docker container

If you run docker run --rm -it alpine:3.7 /bin/sh, created docker container will use default bridge network.

You can see the properties of this network using the following command: docker network inspect bridge. You can see there to which host interface (usually docker0) the network is associated and also what is the subnet for this network (usually 172.17.0.0/16).

Additionally, if you run docker run --rm -it alpine:3.7 ip addr command, you should see the IP address of your container, which should be part of previously mentioned subnet.

Then, Docker by default creates NAT iptables rules, which allows the containers to communicate with outside world (as the container IP address is not known to the internet). You can see the rules using sudo iptables -t nat -nvL POSTROUTING command. It should look similar to:

Chain POSTROUTING (policy ACCEPT 144K packets, 15M bytes)
 pkts bytes target     prot opt in     out     source               destination
  392 75916 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0

Finally, for your container, there is no difference, if you try to reach remote site of your VPN or the internet, as the container only knows how to reach Docker network. You can see that using docker run --rm -it alpine:3.7 ip r command:

default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 scope link  src 172.17.0.2

So when traffic from the container goes via default route, it hits host routing table (I'm not 100% sure if this is right), it is being routed to your tunnel and then iptables NAT rule applies, changing the source IP of your traffic to IP of your end of the tunnel, so returning traffic will come back over the tunnel as well and then back to your container.


The answer depends on how you want to run the container.  There are 3 ways you could accomplish this:

Since you mentioned creating a cluster, you could create a new Kubernetes cluster and run your containers in this cluster.  AWS does support creating a cluster and managing of it via the CLI.
You can choose to run the containers as part of AWS Elastic Container Service (ECS).  This supports launching the containers via AWS Fargate (essentially a "serverless" type approach where you pay for CPU and Memory consumed) or on a standard EC2 instance.  ECS has its own CLI (separate install).  ECS will also allow you to set the ports and set inbound/outbound rules.
Finally, you can choose to use the CLI to spin up an EC2 instance and then manually or via a script start the docker containers.  However, you'd be much better off choosing one of the first two options.

Finally, AWS does have a tutorial that seems to go over exactly what you are trying to accomplish (configuring a container to run via the CLI).


What is the best way to do REST with puppet?

While it may not be the answer you are looking for, puppet's DSL lacks verbs on purpose.  Puppet's idempotent philosophy is a huge reason it is so useful for some use cases but for this type of need you will want to use a tool which is more targeted to that role.

When you hit limitations due to the idempotent philosophy of Puppet it is a good indication that another tool is required for that need.

I would look for publicly maintained tools for the task maybe in Ansible or Terraform.

We leveraged the "puppet kick" functionality of puppet and then tried to extend it after it was deprecated and the pain of trying to ignore their design philosophy is an expensive one.


I do not believe that there is a way to set this as a global policy.  However, there is a simple command you can run that will "prune" or remove your unused images.

You will want to use docker image prune --filter [options] to achieve this.  Check out the official documentation for information on how to best filter.  As a best practice you should be tagging your images which will make managing of them easier using tools like this.


How to use Jenkins to manage a repository from Github with a submodule from Bitbucket?

It is a convenience that Jenkins will pull from git for you. With such a complex setup as needing to pull from two repos you can simply use a pipeline job that runs "sh" to explicitly git checkout the code and pull the submodules:

sh "git checkout ${custom_env_var} ${custom_parameter}

The good news with this slightly manual approach is that you can debug the sh commands to run on a dev server and cut paste them straight into a pipeline. Just start a “sh” session (usually it runs bash) and test the commands.


Your next step will be to identify a solution for container orchestration.  For a quick dev test solution, you could host your own docker registry, push your built image to the registry, and add a deployment step that stops the existing container and spins up a new one from your registry.
However, in a production setting there are many other aspects to consider such as downtime, versioning, security, etc.  There are several products and tools that are designed to handle containers such as AWS Elastic Container Solution (ECS), Kubernetes.


Making thousands of microservice/dockers with different IPs

Linux, without docker

In Linux, you can create arbitrarily many IP address for each interface. A comment in https://serverfault.com/questions/328146/max-number-virtual-ip-addresses-per-nic reports success with 2000, and hearsay of 5000 successful IPs.

So. Pick any Linux box on your intranet, which could of course be a VM, and create as many IPs on its single ethernet interface (MAC address) as you wish.

Say you're using the 10.0.0.0/8 private network, then you can execute lots of these:

ifconfig eth0:1 10.0.0.2 netmask 255.0.0.0
ifconfig eth0:2 10.0.0.3 netmask 255.0.0.0
...
ifconfig eth0:260 10.0.1.6 netmask 255.0.0.0
...

You should be able to use curl --interface eth0:3 (or maybe --interface 10.0.0.3) to use a specific source address. For wget, it's --bind-address=10.0.0.3 etc. Start the curls in the background ("&") and they will run more or less concurrently - or look for some test driver (different question) to run them for you.

This will give you lots of requests from separate IP addresses, but all conveniently running on one machine.

Docker, trivially

With docker: just make an image, as small as humanly possible, which connects to your server. You can use curl without any special options.

Then fire up your 1000 docker containers, just like you usually would. Each will automatically get an IP address on the internal docker network (172.x.0.0/16 by default).

https://stackoverflow.com/questions/21799382/is-there-a-maximum-number-of-containers-running-on-a-docker-host indicates that it is possible with hardware that is not too uncommon today; a naive 1000 docker containers used roughly 3 GB of RAM for them. That question also has some hints on options you can use to decrease the memory needs. Aside from memory, there won't be too much overhead, i.e. CPU wise it should comparable.

Docker, with special networking

docker network opens up manually created networking for you. You should be able to cobble something together (just check the individual --help pages, it's all pretty self explanatory), and combine that with the ifconfig approach explained above. This would mean that you only need one container for your curls, and start 1000's of them inside that container.

I would try to just run your 1000 containers, first, though, unless you are limited in the RAM department, for ease of use.

About the curl...

Whatever you do, don't just start a single curl in each instance, but have each runner be an endless loop of back-to-back curl calls. This way, you avoid long gaps inbetween where your individual jobs are being started and stopped; this is obviously especially important for the "1000 docker containers" variation. In the answer linked above, they note that it took them some good amount of minutes to start up those thousands of containers, so you do not want to do that over and over again.

And you might as well get rid of curl altogether and create small networking app in the language of your choice (Perl, Ruby, C, Java...) to avoid process startup overhead.


Speaking primarily from the Google App Engine (GAE) perspective.

My preference is for environment-driven project split. This might be of interest: Advantages of implementing CI/CD environments at GAE project/app level vs service/module level?

I wouldn't go for architecturally-driven project split unless there are serious reasons for doing so (if you feel you have such reasons you should ask with details):

probably difficult (if not impossible) to avoid inter-project bandwidth charges
added configuration complexity for inter-operation between projects (GCP pretty much considers them completely isolated/standalone, except maybe for shared IAM permissions). Possibly relevant: Google app engine access cross project resources

Depending on the GCP products used there may be sufficient support for architectural splits inside the same project. At least GAE does IMHO.

My GAE apps are all standalone and do not use inter-networking at all, so no costs from this perspective. There are still networking (bandwidth) costs for accessing various GCP-provided infra/services, but those are irellevant/unavoidable from the project split perspective.


What are the ways to get authorized access to mlflow through Azure Active Directory?

It looks like MLFlow doesn't natively support any authentication schemes and the recommendation is to use a reverse proxy such as Nginx to only forward requests to MLFlow if a valid authentication cookie is provided, redirecting any requests without a cookie or an invalid cookie back to the Identity Provider.

Azure AD Application Proxy

You can configure Azure AD Application Proxy to add authentication, although you will also need to protect MLFlow from unauthorised access by putting it in a virtual network or creating rules to only accept connections from Azure AD Application Proxy.

Used managed ML Flow in Azure

There is also the potential of using MLFlow that is part of Azure Databricks, which probably also includes authentication.

Nginx

There are complete instructions on setting this up on the Nginx for a generic application. In summary, however, you will need to:

configure Nginx as a reverse proxy
prevent access to MLFlow from anything other than Nginx
create an application in Azure Active Directory for Nginx
configure Nginx to look for the JWT token and validate it

You would also be able to use other products such as Google's Identity Aware Proxy and ForgeRock's Identity Gateway.


How to apply using Terraform to launch multiple EC2 Resource with different configs (VPC not maintained by TF)

You can use a remote backend as a data source. That is working well for us thus far but this project is not very mature and we will likely refactor a time or ten.

data "terraform_remote_state" "network" {
  backend = "s3"
  config {
    // some variables related to config would likely go here
  }
}

then for the teams not managing the network to consume that data you could create an instance like (special attention to the subnet_id here):

resource "aws_instance" "oracle_ec2" {
  ami           = "${lookup(var.ami, var.region)}"
  instance_type = "${var.instance_type}"
  key_name      = "${var.key_name}"
  subnet_id     = "${data.terraform_remote_state.network.subnet_id}"
}

The network team should be able to manage the networking infrastructure and you should be able to consume those values based on current state.

Terraform Remote State Data Source

  2) If new instances launch as t2.large(Instance type passed as variable) while old ones run on t2.micro

If you change the instance type from t2.micro to t2.large then yes the t2.micro instances will be destroyed and replaced with the t2.large. The name of the game is immutability outside of meta data changes like altering tags. Look into lifecycle hooks to alleviate some of this pain

lifecycle {
  create_before_destroy = true
}

  will it recreate old instances too if I just increase the count

No, current instances would not be destroyed. Terraform will check the state to see how many instances you currently have. If more instances are necessary, terraform will know how many instances to create and create the instances necessary

  Also, is there a way to automate this process i.e. append something like a template for new EC2 instance to existing .tf file.

Not sure what exactly you mean by this one. What I will say is that having the data coupled to the instance itself makes instance immutability and using terraform more difficult. A (untested and unproven) thought and potential solution in my head is to:

//create an ebs volume
resource "aws_ebs_volume" "oracle-data" {
  ...
}

//attach that volume to the instance
resource "aws_volume_attachment" "oracle-data-attachment" {
  ...
}

with some bash involved for an init script that creates and mounts the new volume to be used in conjunction with the aws_volume_attachment resource. That could potentially create a new instance preserving the EBS volume then reattaching it to the new instance.


What are ways to mitigate the effects of Mythical Man Month?

What is the MMM

First I want to explain the context for Brook's Law. What was the assumption that made him create it back in 1975?

  A Man-month is a hypothetical unit of work representing the work done by one person in one month; Brooks' law says that it is impossible to measure useful work in man-months.

source: https://en.wikipedia.org/wiki/The_Mythical_Man-Month

Back in the day, complex programming projects would mean big monolith systems. And Brooks claims that these cannot be perfectly partitioned into discrete tasks that can be worked on without communication between developers and without establishing a set of complex interrelationships between tasks and the people performing them.

This is very much true in highly cohesive software monoliths. No matter how much decoupling is done, still the big monolith mandates time required for the new programmers to learn about the monolith. And increased communication overhead which will consume an ever increasing quantity of the time available.

But does it really have to be this way? Do we have to write monoliths and keep communication channels to n(n − 1) / 2 where n is the number of developers?

We know there are companies where thousands of developers are working on big projects ... and it does work. So there must be something that changed since 1975.

Possibility to mitigate the MMM

In 2015, PuppetLabs and IT Revolution published the results of the 2015 State of DevOps Report. In that report, they focussed on the distinction between high performing organizations vs. non-high performing.

The high-performing organizations show some unexpected properties. For example, they have the best project due-date performance in development. Best operational stability and reliability in operations. As well as the best security and compliance track record.

One of the surprising things highlighted in the report is the deployments per day metric. But not just deployments per day, they also measured deploy/day/developer and what is the effect of adding more developers in high-performing organizations vs. non-high performing.

This is the graph from that report -

While low-performing organizations align with Mythical Man Month assumptions. The high-performing organizations can scale the number of deploy/day/dev linearly with the number of developers.

An excellent presentation at DevOpsDays London 2016 by Gene Kim talks about these findings.

How to do it

First, how to become a high-performing organization? There are a couple of books that talk about this, not enough space in this answer so I'll just link to them.

Good to Great: Why Some Companies Make the Leap...And Others Don't by Jim Collins
The High-Velocity Edge: How Market Leaders Leverage Operational Excellence to Beat the Competition by Steven Spear
The New Economics for Industry, Government, Education by W. Edwards Deming

For software and IT organizations, one of the critical factors for becoming a high-performing organization is: focus on quality and speed.

For example Ward Cunningham explains Technical Debt as all the things we allowed to be left unfixed. This is accepted by management because it always comes with a promise that it is going to be fixed when there is time.

There is never enough time, and technical debt just becomes worse and worse.

What are these things that cause technical debt to grow?

legacy code
manual configuration of environments
manual testing
manual deployments

Legacy code As defined in Working Effectively with Legacy Code by Michael Feathers is any code that doesn't have automated testing.

Any time shortcuts are used to get the code to production; the operations is burdened with the maintenance of this debt forever. Then the process of deployment becomes longer and longer.

Gene tells a story in his presentation about a company who has six-week long deployments. Involving tens of thousands of extremely error prone tedious steps, tying up 400 people, and they would do this four times every year.

One of the tenets of DevOps is that reliability comes from doing smaller deployments more frequently.

Example

Jon Jenkins at Velocity 2011: Amazon are doing 15k deploys/day
Ken Exner at AWS Summit 2015: Amazon are doing 136k deploys/day

These two presentations show all the things that Amazon did to decrease the time it takes them to deploy code to production.

According to Gene, the only thing that is changed across time in these high-performing organizations is the number of developers. So from the Amazon example, you could say that in four years they increased their deployments ten times just by adding more people.

This means that under certain conditions, with the right architecture, the right technical practices, the right cultural norms, developer productivity can scale as the number of developers is increased. And DevOps is definitely in the middle of all this.


Why trigger something remotely (using Jenkins) and then just forget about it?

I wouldn't describe Jenkins jobs as "fire and forget."  They are designed to run automatically without the need for human interaction, however, that doesn't mean the result of a job is not visible.  Quite the opposite- detailed logs of all activity are available for jobs ranging from command-line tools to custom deployment jobs.

The testing results from Jenkins jobs are routinely displayed on large-screen TVs all over the Development floor of many companies I've seen.  There are whole plugins devoted to making this type of information available to developers, Ops, and management in a dashboard format.


Ansible dynamic inventory using AWS plugin -- how to obtain internal DNS records or private IP?

The hostnames parameter controls what the plugin uses as the host name. It accepts a list of attributes, and will use the first one that the instance has defined:
hostnames:
  - private-dns-name
  - private-ip-address

You can also set the ansible_host variable using compose. This changes the connection target without modifying the inventory hostname:
compose:
  # Note the underscores; this functionality uses the Ansible variable names instead of the raw AWS attributes.
  ansible_host: private_ip_address


Strange orchestration behaviour of a Docker Swarm manager - pile of containers

I would check that all nodes are active and ready with command

docker node ls

With

docker swarm leave

and

docker swarm join \
--token <SWMTKN token> \
<manager private IP>

you may rejoin malfunctioning node.


How to mark a step as failed or unstable on timeout, instead of aborting build?

I've done something similar to this in the past:

stage('Step Tests') {
  steps {
    dir('test') {
      script {
        try {
          timeout(time: 5, unit: 'MINUTES', activity: true) {
            sh "yarn step-tests"
          }
        } catch (Exception e) {
          currentBuild.result = 'FAILURE'
        }
      }
    }
  }
}

I haven't tested this code exactly as I wrote it here, so think of this more as a kind of rough sketch rather than polished final product.


Run docker applications within a kubernetes cluster

After reading your Q&A I think that you are looking for a message broker like kafka, rabbitmq or Amazon MQ. Please let me know whether this helps.


Load Balancing of Jenkins Nodes without Load Balancing Plugin?

This is similar to:
https://stackoverflow.com/questions/55023570/how-to-run-a-job-on-first-available-node/55567958

Have you considered Least Load plugin?

https://plugins.jenkins.io/leastload

By default Jenkins tries to allocate a jobs to the last node is was executed on. This can result in nodes being left idle while other nodes are overloaded. This plugin overrides the default behavior and assigns jobs to nodes with the least load.

Don't imagine there's a security issue there.


I don't know of a way to get the actual GCE VM images.

But it's possible to export the images to GCS (as a tar.gz archive):

  If you need to move your Compute Engine boot disk data outside of your
  Compute Engine project, you can export a boot disk image to Cloud
  Storage as a tar.gz file.

  You can export a custom image as backup or for sharing by exporting
  the image to Cloud Storage. This method is ideal for sharing
  individual images with other projects that do not have access to your
  images.

You can then download this tar.gz file and, I guess, create a VM image from it to run on your local machine. I imagine the process would contain steps similar  to those detailed in Migrating VMs to Compute Engine and Importing Boot Disk Images to Compute Engine, only seen from your VirtualBox perspective instead of the GCE one.

These guides are making it easy to transfer VM loads to GCE. I guess they don't have incentives to do the same for transfers from GCE, which probably explains why there is no guide for it ;)


Kubernetes expose an app via a DNS name in minicube

The approach I've seen work for this is to use Nginx as the LoadBalancer and defining the Ingress with type NodePort for the different services.  You can see a tutorial that shows the first steps for setting this up here.

Additionally, the official docs DNS for Service and Pods and Customizing DNS Service are good resources.


How to report Code Coverage metrics across git repositories?

If you use python and GitLab like I do, you can include a test coverage report in your CI/CD pipeline. You pip install coverage and then just run coverage in your pipeline. The following is an excerpt from an AWS lambda pipeline, but you should be able to find something similar for your environment:

 script:
    - pip install -r ./awslambda/requirements.txt
    - coverage run -m unittest discover -v -s "./awslambda" -p "*_test.py"
    - coverage report

The above generates a nice code coverage report and it is free of charge, for Python, AWS and GitLab.


Boto3 is a client for AWS API, so by definition it doesn't handle the synchronous or asynchronous behavior of the API call, that's the API endpoint which define that.

You can safely assume that any API endpoint marked as asynchronous will be asynchronous in any API client. Some SDK implement methods to wait for those asynchronous calls, some don't and you need to write your own handling (usually polling regularly the state) if you need to wait the end of a task started by an API call (creating a CloudFormation stack for exemple).


Jenkins: Re-use repository with reference option and do changes on it

If this is not possible is there any other way without duplicating the 100GB for each jenkins job?

To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path.  Then, in your Jenkins job, you can cd to that directory with the dir step, update the repo, and checkout the branch you want, e.g.:
dir('/path/to/repo') {
  sh('git fetch --all')
  sh('git checkout origin/lineage-16.0')
  // your build steps here
}

However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption.  It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.
You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.


Terraform error Reference to “count” in non-counted context for resources not necessarily data

looks like your new to stack exchange. You need to form a clearer question to get a good answer....

As far as your error goes.... count is a reserved name. It's only accessible when the block you're writing has a count set!

You have You just need to specify the count of aws_internet_gateway data resources you're trying to get. Your code implies there is more than one.

data "aws_internet_gateway" "zyz_igw" {
   count = 1000000000
   ...


AWS Redshift UNLOAD not firing S3 PUT event

You did not tell how big data amounts you tested with, but if you test ok with little data block successfully and larger real situation fails, there may occur same as occurred here:

  The larger files are almost certainly being uploaded via S3 Multipart Upload instead of a regular Put operation.

  ..

  It sounds like the function is only subscribed to s3:ObjectCreated:Put events currently, and you need to add s3:ObjectCreated:CompleteMultipartUpload to the configuration.


Is there an ansible module that check whether a port is not in LISTEN state?

I think you're looking for the wait_for module. It will allow you to check and make an action when a certain port is available and more.


Active directory integration issue with Jenkins

We had to mention the specific DC on the Site field


How to read array and loop server selection and deployment

Look this:

#!/bin/bash
#Get servers list
set -f
string=$DEPLOY_SERVER
array=(${string//,/ })
for i in "${!array[@]}"do
      echo "Deploy project on server ${array[i]}"
done

I found this link


What does it mean that "outside of memory-managed code execution runtimes (JVM, Go, etc.) the usefulness of unikernels starts to rapidly decline"?

Well, I'm not a big expert on the matter and, in fact, I don't understand the statement. I've read it several times and there is no further explanation, so I cannot say it's make sense or not for me because I just don't know where the idea comes from.

Anyway I'll try to expose what I think here.
The main difference about managed and unmanaged code is who is in charge of the memmory management. If it's the developer, we're talking about unmanaged code (C language). If there is a garbage collector or any other similar mechanism, it's managed (Go, Java, .Net platform, Python even bash if you want to consider it!!). So I don't undertand how important is who reserves/frees/whatever the memmory.

And, IMHO, pros of unikernels are supossed to be:

Performance.

Kernel modes are supressed (rings for x86, modes for arm, ...) and the application that will be running in the system, as well as any library needed, is compiled and statically linked into the kernel and the overhead introduced by these mechanism (syscalls basically) are supressed.
Kernel is reduced as much as possible. No USB, bluetooth, support for lots of FS and unnecessary code is just not compiled. Kernel gets tiny.

Security. This is a sensitive topic. Again, I think unikernels are safer than the traditinal ones. If it's a well-done job, there won't be nothing than the minimun functionalities implemented into the system, so this should reduce possible break-points. However, if any is found, this will be more dangerous. Every single process is running by the privileged privileged (and only, multiuser capabilities are disabled in unikernels) user in the system. Privileged Remote code executions can happend. However, it should be in a container-like system. You just can restart it and (in no persistent systems) all the work done by the attacker should be gone (but still an upgrade should be needed...).

As I said, I cannot figure out the reason of this statement, but I think it's related to performance rather than security. In that case, I'm wondering why should be a performance improvement reducing syscalls compared to containers if:

Containers share the running kernel within the host where are running, so syscalls are done just once as there is just one instance of the kernel running.
Unikernels, as said, do not use syscalls and every procces runs in privileged mode (can access the harware directly). But, in this case (I THINK) as far as kernels are different, you need a virtualization layer, so total running kernels are 2. Unikernel has no syscalls but the host kernel does, so syscalls are not supressed and will run once. Of course, I'm assuming that no-one will run a unikernel system on a bare-metal.

So, the number of syscalls should be pretty similar to the containers scenario. Isn't it?


Kubectl: the server doesn't have a resource type "cronjobs"

The answer is appearently that there's a bug in Minikube 0.26. Downgrading the Minikube installation is solving it.


Where would DevOps Engineers mostly look for their next opportunity and what would it be?

Understand that the term DevOps refers to both a Strategies and Tactics.
For instance, the strategies of Continuous deployment and Consistent development/production environment.
The tactics may be using Kubernetics for deployment to the cluster with Docker for development/production environment.
With that said, a DevOps shop will likely use Kubernetics and Docker as their tactical tools and would look for people with those skillsets.

ALSO, they would look for people who are a CULTURAL fit who would work well in a collaborative environment.  If your background is let's say operational and your not open to learning programming skills(or learn what the programmers need), don't take ownership of a project or work well in a team...you will NOT be a fit for the company (you won't get hired).

As far as the best place to find a DevOps related job?

I would look at going to a DevOps Meetup, go to trainings/seminars where DevOps tactics and tools are taught (i.e. Docker, Vagrant, Ansible, Kubernetics, etc.), look for certain keywords in the posts such as SRE (Site Reliability Engineer), Systems Automation, etc... and do some research on companies that are moving to or have embraced DevOps Strategy.

If you understand these things, I believe it will give you a great advantage at landing a job in one of those companies.


What is the common term for Apache virtual hosts or Nginx block systems?

All my searches for a term brought me to conclude that the general term either for one Apache virtual host file or a constellation of Nginx block files (each containing at least one block) is just that --- a virtual host.

Per project; either the virtual host (entity) is a single file or a combination of files, and that's true if the webserver software is different than these two.


The scheduler itself is not swappable. However, you can set memory constraints and reservations on containers in your service:

version: '3'
services:
  redis:
    image: redis:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 50M
        reservations:
          cpus: '0.25'
          memory: 20M

The constraint will kill the container if it exceeds the limit even if the host has more memory available. The reservation will ensure that amount of memory on the host is available for the container.


One Checkout When Running Parallel Stages In Jenkins

Your difficulty stems from encouraging potentially long-lived branches, which is deviating from the CI methodology - which is supposedly why you'd be using jenkins - a CI tool - in the first place.

Normally you shouldn't be running the tests for those branches in the Jenkins pipeline for the trunk - you want to measure the quality of the integrated work, not that of the work done in the silos.

I do understand the desire to run such tests before merging into trunk, so maybe it's OK to use jenkins to do it if you want to, but with separate jobs/pipelines, not tied into the trunk pipeline. And keep in mind that the results of such tests aren't necessarily conclusive: just because they pass over there doesn't mean the same tests will also pass in the trunk, you'll need very strict conditions to make that always true.

Personally I'd go for a fully automated patcing/merging solution based on pre-commit verifications which can enforce the strict conditions necessary to prevent post-merge trunk quality regressions (but I'm biased, I built such solutions). See for example How to ensure that git subtrees are kept up to date?

Anyways, back to the question. Since the code for the various pipeline segments that you seek to perform in parallel comes from different branches you won't be able to use the same workspace, you have to pull separate workspaces from the respective, different branches. Which makes the question kinda moot.

Final note: assuming the same workspace could be used in multiple builds, your #2 approach requires a pristine build structure. It's not uncommon for less-than-perfect builds running in parallel in the same workspace to interfere with each-other: race conditions creating/deleting directories, overwriting artifacts, too high peaks of resource usage, etc - typically hard to repro intermittent failures. These could easily happen if, for example, the builds would be for the same product but for different CPU architectures, or builds including code shared across multiple products. Just be aware of the possibility.


Spin up an EC2 instance from within the application

Use AWS API to spin up the instances. There are SDKs (Software Development Kits) for various languages, for example if you're using Python use boto3 library. SDKs for other platforms are also available: Tools to Build on AWS, including the command line tools AWS-CLI for use in shell scripts.

With the SDK of choice you'll simply call something like ec2.start_instance(...) to spin up your new EC2.

However I would question whether you really need standalone EC2 instances for your processing? Perhaps ECS Containers, especially Fargate, or even serverless lambda functions triggered from a work queue may be a better choice?

Hope that helps :)


How can I ensure consistency between new microservices?

I'll add an answer of what my solution is so far, but I'm still really interested in hearing how other organisations are solving this problem and the best practices they have.

To resolve the issue of not having a consistent base to create projects for, my idea is to create a repository (repositories?) of boilerplates/templates and use cookiecutter as a tool to scaffold out new microservices. This way, each project is created from a standard base with all the lessons we've learned as an organisation integrated in to it. Any changes we make can be integrated upstream in to the boilerplate repository. I imagine we'll have templates for Nodejs Docker images, Serverless SPAs, Python lambdas, etc.

To resolve the problem of changes made to the templates being propogated downstream to every project, my solution is to implement a process where owners of microservices are made aware of changes to the template and are then responsible for propagating those changes to their microservice.


Which Chef software do we need to install for Windows?

A node in chef terminology is a machine managed with chef-client.

Chef-client is available for various OS, windows, Linux, Aix, Mac etc.

However the chef-server is only available for linux.


helm - cannot get resource "namespaces" in API group

Check https://github.com/helm/helm/issues/3130, this might help.
I followed the instruction in the post:

kubectl --namespace kube-system create serviceaccount tiller

kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

helm init --service-account tiller --upgrade

This works for me.


User data in original windows ec2 instance does not get executed in custom ami

User Data Persistance

If I create an image with Packer or other tool, I'll include normally a userdata initialization script that would allow connecting via WinRM. This is required as the default AMI settings wouldn't allow the tooling to connect. In this scenario Packer would create the instance from an AMI with my provided userdata.

However, when I then take an AMI of this configured instance and try to use it as a new instance, this new image has it's own userdata. If I wanted to run the exact same command, I would need to create the userdata in this new instance.

Userdata isn't persisted in your AMI, it's more an instance level configuration value.

Follow-Up Comments on This

The goal should be to create your AMI with the tooling, for example git, and then your new instance should not even need to run that command in the first place as you've "baked" it into your "golden ami". Consider this in your approach. Also consider decoupling any installation directly from userdata and use something like AWS Simple Systems Manager Command documents to run the installation. This would decouple your installation code from your infrastructure allowing you to apply automatically as well as on demand without having to repeat your work.

User Data Content

The code example above has a typo in the closing tag for PowerShell. I'd try replacing this and see if that's the root issue. If there's further issues after fixing that, post a comment on this answer and I'd be glad to look further.

Also as far as powershell syntax consider using join path for proper path parsing.

For example $exe = Join-Path $ENV:ProgramData 'git/git.exe

The. You can run

&$exe --config --global ....

I'd also recommend that you consider looking at Amazon systems manager documents. You can create a  document that initializes a setting such as get config.

Then you create a AWS association to this document on a tag, at that point it will run automatically anytime a new instance is created without you having to continue to maintain it. It also allows you to update the document independently of the user data, and then ensure you can run it manually as needed as well. I'm a big fan of decoupling any type of configuration scripts from the userdata and using that as a last resort.

One last item...

Remove-Item C:\c -Recurse -ErrorAction Ignore

Highly recommend you revisit that. That's seems super easy to mess up and start deleting stuff you didn't mean to. I'd make sure you use a fully qualified path with Join-Path and also use the -LiteralPath instead of positional argument by default. If anyone created a folder c on c drive I'd reject that in a heartbeat in a code review.


Sprints section shows tasks as unparented

I was able to contact DevOps Support and get an answer. The issue was that I could only create Portfolio Backlogs, not Requirement Backlogs. By changing the default Requirement Backlog to include my custom work item, those showed in the Sprints section.


Installing phantomjs karma launcher in Ubuntu-Docker context

Because fsevents is an optional dependency, the solution has been here:

npm install --no-optional karma-phantomjs-launcher


You're basically describing GitOps. Here's how I've followed that for a containerized microservices build and deploy pipeline:

Each service repo includes it's own Dockerfile, Kubernetes manifests, and anything else needed to locally build and test the project. The base image for the Dockerfile will likely be managed from another repo but built and hosted by the organization.

CI steps are mostly externalized to a set of common libraries or templates. So if there's a Jenkinsfile or equivalent, it sets a few variables and calls the common template to build and generate the artifacts.

Artifacts include a docker image, pushed to a registry, and also the deployment manifest (compose file, Kubernetes manifest, Helm templates), along with any other variables for those manifests. These deployment manifests/variables are pushed to a separate deployment repository that includes all of the microservices' manifest. There is only one deployment repo for all of the microservices in a single application stack, so this commit becomes a fan-in step. The push to this repository is scripted/automated during the CI of the microservice repo when a pull request is approved by enough trusted people to a protected branch. This provides the needed security checks.

The deployment repo commits triggers it's own pipelines that are effectively CD and following something close to a GitOps design. This could also be a GitOps tool monitoring the repo and applying the changes, but presently I've been sticking with a single CI tool that does both, simplifying the workflow and allowing some scripting to run on the deploy. That scripting for me loads the variables that were committed, applies additional variables (e.g. which environment is this being deployed to), renders the templates (currently a Helm umbrella chart), and deploys.

Deploys to other environments becomes a pull request between branches of the deploy repository. It could also be separate repositories to enforce security controls. Importantly, rendering the manifest should point to an immutable image tag or sha hash to ensure what was pushed to dev and tested there is the same as what's pushed to production. I include these image tags in a values file that is committed to the deploy repo.

One big advantage of this, multiple microservices that interact with each other are tested together and get deployed together as a unit, all services are fanning into a single deploy repo that contains their deployment manifests and what image tag to use. There's never the situation where service A gets a new feature, but it's buggy, service B gets built depending on that new feature, and service B gets released to production before service A (a common issue when each service is built and deployed in isolated pipelines). If A is blocking the deployment of the stack, then a pull request can be submitted to the deploy repo reverting the service A commit. Then if service B passes tests against an older service A, service B gets deployed to production.


Migrate Rancher host from server to server

First of all you need to support HA mode. You will need:

external MySQL DB (e.g. AWS RDS Aurora)
external load balancer (e.g. AWS ELB)
2 additional Rancher Server Nodes (to support quorum of 3)

Once you will be ready with external DB Rancher would ask you to backup & restore his database in it. After that you'll be asked to redeploy Rancher Server with additional params for external DB connection. Then you'll be able to run two more Rancher Server Nodes (e.g. using AWS ELB).

When previous steps have finished you would be able to add more Rancher Agent Nodes in AWS directly from Rancher Hosts tab. After that you can just switch off OVH nodes - payload would be transferred to AWS automagically.

Make sure to backup & restore all databases or data-sensitive instances on OVH hosts.

More information such as HA Requirements etc. can be found in documentation.


What is the best way to achieve DevOps in any environment, or it is always environment dependent?

The answer to this question about what is DevOps?

DevOps both involves Strategy and Tactics.

Strategy embraces building a culture centered around collaboration in your software development, technical operations and any other group or organization that has a stake in the deliverables whether it's software or service.

Examples of DevOps strategy may be breaking down silos, continuous delivery, continuous integration, etc.

Tactics are the actual implementation of the strategies.

This is the level where the use of software tools, technology and management methods are found as well as the actual "how to".  For instance, to foster "breaking down of silos" as a tactic, you build smaller "agile" teams of 5-7 "stakeholders" let's say programmers, systems admins, QA engineer, and a project manager.  To communicate among the team members, you use Slack.  Programmers and sysadmins "cross train" on each others tasks where it makes sense.  Sysadmins may assist in developing test harnesses for the programmers, use Selenium and python for testing.  Programmers learn how to use Ansible and assist the Sysadmins on what they need in the development and production environment.

Where you sound like your at is looking at all these cool tactics and tools that DevOps uses such as Docker, Ansible, Chef but your missing the understanding of Strategy...building of the culture, the "why" you want to use these tactics and tools.  That's ok, we're here to learn.

There is no "right" answer on what exactly DevOps is or how you implement it, but in all the talks and examples of DevOps shops and organization it boils down to the strategy of building a collaborating organization and focusing on a purpose then put it into action as tactics.


How to filter by entity name using sensuctl?

Using some shell commands in cmder to parse the json and filter the results I was able to filter the results. Here is the command:

.\sensuctl.exe dump entity --all-namespaces --format wrapped-json | jq ".metadata.name" | grep -i <filtertext>


Mitigating Maven Central risks as seen from the DevSecOps perspective

What could be a feasible way to motivate Maven and repository makers to establish some kind of early time warning, like issue special headers on download, or and additionally for example reject download of vulnerable components by default?

Your concern, i.e. the meat of your question is a very valid one: how to improve security when downloading "stuff" from the internet. Maven, yes, but also other places like Dockerhub, Debian, Ubuntu etc..

I have no idea what stance the Maven developers have on this; I assume they have thought about it and come to the conclusion that it is not a good idea, for whatever reason. Generally speaking, I see no particular reason to place this responsibility on the repository, though.

Having separate responsibilities between componentes (here, Maven Central as one component) is a very powerful concept in IT. You find it in Unix command line tools, in OOP class design, or server architecture. In this particular case, it seems perfectly valid to me to let Maven Central simply be a huge repository of "stuff". Its main function is to be there and supply the packages.

Placing a security filter, if you will, between mvn and Maven Central is then pretty easily achieved. For example, a company I work for runs their own hierarchy of Nexuses; while the developers can pretty much do whatever they want on their laptops, the automated build environments have no internet access and must go through curated Nexuses. Each project has their own Nexus (for seldom used packages), and they chain up to a common intra-company global Nexus.

Developers, then, have no way to insert packages into those Nexuses, but a dedicated team reviews new packages and decides whether they can be used.

Note that this extends not only to security, but also, for example, to licenses. Depending on the policy of the company, not every open source license may be acceptable. So this can also be screened for.

Another big reason would be non-security related bugs. Say you find that X of your projects use package Y, and Y is found out to contain a serious bug, maybe with the result of being able to trash all your data. Would Maven Central remove that version from their repository? Certainly not! The bug may only appear in a very small edge case, which, by chance could be very relevant to your company. So, by having your own repository structure inbetween, you can flag the affected versions as "invalid", and thus enforce bug fixes traveling downward to your projects.

Finally, a company might want to phase out old versions of packages as a matter of routine, simply to keep up with the times. Using such a "filter" approach helps with that as well.

Back to Maven Central: why do they not do it? The question is not just a technical one, but also organizational. Who decides what is a relevant breach of security? Are we only talking about actual bugs? Are we also excluding packages that are safe in the hands of a responsible and knowledgeable user, or do we also by default remove packages that lead to possible holes indirectly (i.e., in a Unix package management, one may designate the telnetd package as insecure although it is, per se, perfectly fine).

So there you have it. Such filtering is a complex issue, and complex stuff is best distributed to different entities instead of forcing it on one central body.


As you guessed, build not necessarily means "compile". Building is a process of creating software from sources, which might include stuff like gathering dependencies, compiling, archiving, transforming code in any way, but also testing, deploying to different environments and promoting the artifacts between them.


Is there an established metric to describe optimal/needed ressources for a specific computational task?

As far as I'm aware, there is no universal standard amongst cloud providers. One reason for this is because most cloud providers like to abstract how their backend infrastructure, that their instances run on, is managed. How your instance is ran could change minute by minute depending on the cloud provider. The user doesn't care as long as they are getting the desired computing power that they want. Cloud providers also may not want to divulge their exact infrastructure metrics for proprietary reasons as well.

Despite these reasons, some cloud providers offer a standard amongst their own virtual instances. These standards are relative to the computing power of each instance type that the provider offers.

For example, Amazon Web Services uses EC2 Computing Units (ECU) as a metric to compare the approximate computing power in relation to each instance. Comparing two instances types such as an m5.large and an m5.xlarge for example, the m5.large has an ECU of 8 vs the m5.xlarge which has a ECU of 16. The m5.xlarge has roughly double the compute power and therefore double the cost.


Release based on restricted timeline with CI/CD

Branch-by-abstraction and feature-flags/feature-toggles are the typical techniques allowing development of any combinations of features in various degrees of readiness in the same branch. For a proper CI/CD approach that should be the main/master branch. Each feature being developed can be tested independently by turning the respective flag on or off. "Releasing" the feature in that branch simply means that the respective flag is enabled in the source code

To be effective, all the relevant combinations of the feature flags' values should be subjected to the respective CI/CD QA verifications in this branch. The complication, of course, is the number of such combinations which can cause cost/time "explosions" in these verifications, especially in large scale projects with many features.

Grouping multiple feature flags under a single "super-flag" is an effective method of reducing the number of such relevant combinations. Turning on (in the code base) these super-flags are often time-driven according to a release schedule. Which can be either periodical or based on certain specified/seasonal time/dates, it doesn't really matter (as long the respective features meet the expected quality level by that time, of course, but that's a different discussion).

Turning on the super-flags is all that's needed for the software products that are continuously deployed. But for the products which can't be continuously deployed (like OSes or most of the embedded software products) release branches in which only the super flag(s) (or directly feature flags) applicable to the respective releases are enabled is the typical way to keep the release time/costs under control. They can also serve as the CI/CD branch vehicles for subsequent hot fixes, service packs and/or minor releases.

Release branches would be pulled off the main/master branch (or parent major release branch) whenever the quality level is good enough for the very next release and trouble to support the delta to the 2nd next release remains acceptable (as soon as the branch for the next release is pulled the parent branch can diverge, heading towards the 2nd next release).

And no, release branches aren't (necessarily) evil: they are not feature branches and they can completely avoid any massive branch sync/reparent/merge operation, thus remaining true to the CI principles.


Azure Pipeline PowerShell@2 Task can't run 'invoke-build' command as inline script (build PowerShell Module)

I'm not familiar with Invoke-Build, but appearantly it's a part of the InvokeBuild PowerShell module? In that case, you'll have to install the module as part of the script where you're using it. Your YAML should be something like this:
- stage: Build
  jobs:
    - job: Build
      steps:
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: |
              Install-Module InvokeBuild
              Invoke-Build build

As @Michael Erpenbeck said "the hosted agent starts off as a clean slate". That is to say that it starts off as a clean slate for each job in your pipeline. I'm not sure what bootstrap.ps1 is supposed to do so I can't really comment on whether that script and your Invoke-Build should be run as steps in the same job. But the main point to remember is that any requirements beyond what the standard Azure Pipelines agents offer should be taken care off within each job.


Jenkins declarative pipeline Warning: JENKINS-41339 probably bogus PATH

This post explains that/why environment variables don't propagate to Docker agent.
If you need a custom container, you could define path and whatever else as layers on top of base image in a Dockerfile. Then using that image in pipeline would be what I think is usual way of doing it. You can build that image outside of Jenkins, but it's also possible to build container from within pipeline too if you want (get Dockerfile from SCM or some other use case).
In the context of your question, I think you can still use base image, and if you know full path you'd like to set, do that at container (agent) startup with args. But $PATH variable won't be available at host to be resolved within container at startup. If you must dynamically set path in container, could adjust it in shells.
pipeline {
    agent {
        docker {
            image 'python:3.9'
            // cannot resolve $PATH var in container (becasue HOST does not know what it is).
            args '-e PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/i/know/my/full/path/ahead/of/time'
        }
    }
    stages {
        stage('Test') {
            steps {

                // as set by args above. Applies to all shells
                sh 'echo $PATH'

                // You can also adjust it within container in each shell
                sh '''
                    export PATH=/even/more/path:$PATH
                    echo $PATH
                   '''

                // But this is back to what's in args in a new shell instance
                sh 'echo $PATH'
            }
        }
    }
}


There are no fundamental technical issues with running multiple jenkins slaves on the same machine. In fact Running Multiple Slaves on the Same Machine lists several good reasons for doing it:

  While the correct use of executors largely obviates the need for
  multiple slave instances on the same machine, there are some unique
  use cases to consider:

  You want more configurability between the configured nodes. Say you have one node set to be used as much as possible, and the other node
  to be used only when needed.
  You may have multiple Jenkins master installations building different things, and so this configuration would allow you to have
  slaves for more than one master on the same box. That's right, with
  Jenkins you really can serve two masters.
  You may wish to leverage the easiness of starting/stopping/replacing virtual machines, perhaps in conjunction with Jenkins plugins such as
  the Libvirt Slaves Plugin.
  You wish to maximize your hardware investment and utilization, at the same time minimizing operating cost (e.g. utility expenses for
  running idling slaves).

In general scaling out is preferred, primarily because ability to scale up is typically limited by the types/sizes of the available physical resources.

In particular for augmenting build power I'd recommend an analysis of your actual build  to determine how it uses the machine resources, which/where its bottlenecks are and what scalability limitations it raises to reveal if scaling up even helps.

For example I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially (non-overlapping) on the same machine. In such case I wouldn't even consider scaling up as it would actually decrease the overall building capacity.


Since Ansible was installed via Homebrew on macOS, the workaround is to install Ansible using Pip.

$ brew remove ansible
$ pip3 install ansible
$  pip3 list | grep -e ansible -e Jinja2
ansible      2.8.0
Jinja2       2.8
$  ansible-playbook check_jinja.yaml -v
TASK [jinja_version]
ok: [localhost] => {"changed": false, "msg": "2.8"}


Cannot Change Docker Size on Ubuntu 16.04

The only log message there is "Start request repeated too quickly." so we're missing the beginning of the error. You can run as root systemctl reset-failed and then systemctl start docker followed by journalctl -u docker --no-pager -n 50 to show the last 50 lines of logs.

Most likely, your problem is that your storage driver is not device mapper, and this option only applies to device mapper. On an 18.03 install, ubuntu will be overlay2 with a recent enough kernel, or possibly aufs on older kernels. Without using the device mapper storage driver, you should not be setting that option.


Let me answer each your question in parts.

What the point is?

As per Orbs Introduction:

  orbs are shareable packages of configuration elements, including jobs, commands, and executors.
  CircleCI provides certified orbs, along with 3rd-party orbs authored by CircleCI partners

The advantages of this approach are that an authorized package of jobs, commands, and executors is offered.

Assuming hundreds of orbs, it looks like a community-approved plugin/addon ecosystem. E.g. Ansible roles or Jenkins plugins or Wordpress plugins

orbs vs shell functions/script

Since orb packages:

jobs,
commands,
and executors,

then it can hardly be replaced by a simple combinations of shell commands/functions.

Seems like unnecessary lockin?

As for lock-in concerns: you are right.

It looks like lock-in. And it is lock-in.

But as I know, there isn't any portable solution than can be easily migrated from one CI/CD system to another. E.g. try to convert .gitlab-ci.yml to travis.yml.

But lock-in's, as usual, tend to give you an advantage in speed or ease of deployment.

Portability

Since there are four executor types (docker, machine, windows and macos), it gives you more portable and cross-platform approach.


A name for testing software migrated to a new environment against original application

As mentioned in the question, there is a subtle difference between Regression testing and the situation described here.

To paraphrase Wikipedia:

  Regression testing ... is re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change. If not, that would be called a regression.

However, a situation is described here where the functionality of the code is the same -- since no new features or functionality is added, this is technically not regression testing.

However, what does need to be asserted is the behaviour of the code. If run in the same way after the re-implementation, does the code still behave in the same way, beyond still giving the same results (which would be covered in unit testing and functional test).

In "The Practical Test Pyramid", Martin Fowler describes the various types of testing which a well-maintained application could have. Near the apex are "Acceptance Tests", i.e

  Do Your Features Work Correctly?

After a re-write, I believe Acceptance Tests would be the answer here.

Note - Acceptance tests also include Operational Acceptance Tests, described by Wikipedia as:

  Operational acceptance test (OAT) criteria (regardless if using agile, iterative or sequential development) are defined in terms of functional and non-functional requirements; covering key quality attributes of functional stability, portability and reliability.


Problem with Ansible Asynchronous actions

Order of execution is important Server (Task A) needs to run before the client (Task B)

Then why are your running Task A async in that case ? Is it just because iperf --server is not falling in the background and dying as soon as the shell task finishes ? A quick look at the documentation shows we can probably work around that problem with something like:
iperf3 --server --daemon --pidfile /path/to/iperf3.pid --logfile /path/to/log

Notes:

I added the logfile option since I guess it can be useful but it is not mandatory for the current issue
I took for granted you are running the server on a linux platform. Running it on Windows would apparently slightly change the below scenario. See Running as a service on windows

From what I understand the last unsolved issue is running the client tasks sequentially on each client host. This can be done using the serial keyword on your play.
Here is an untested pseudo ansible setup summarizing what I understand from your requirements. Adapt to your real use-case.
inventory/iperf_hosts.yml
---
iperf_servers:
  vars:
    iperf_pid_file: /tmp/iperf.pid
    iperf_log_file: /tmp/iperf.log
  hosts:
    server1.local:
iperf_clients:
  vars:
    iperf_time: 1
    iperf_dest: some\win\directory
  hosts:
    host1.local:
    host2.local:
    host3.local:

playbook.yml
---
- name: Start the iperf server
  hosts: iperf_servers

  tasks:
    - name: Fire the start command
      command: "iperf3 --server --daemon --pidfile {{ iperf_pid_file }} --logfile {{ iperf_log_file }}"

- name: Launch clients one after the other.
  hosts: iperf_clients
  serial: 1

  tasks:
    - name: Run iperf3 client
      win_command: "iperf3 -c {{ hostvars[group['iperf_servers'][0]].inventory_hostname }} -t {{ iperf_time }}"
      args:
        chdir: "{{ iperf_dest }}"
      register: iperf3_out

    - debug:
        var: iperf3_out.stdout

- name: Now we are over with clients, kill the server
  hosts: iperf_servers
  # We already gathered facts at first play so we disable here
  # You might want to disable in all other plays if you don't need facts at all
  gather_facts: false

  tasks:
    - name: Slurp pid from file
      slurp:
        path: "{{ iperf_pid_file }}"
      register: slurped_pid

    - name: kill the iperf3 process
      command: "kill {{ slurped_pid.content | b64decode }}"


parameter-store values show '***' instead of actual value

The "***" is the masking of parameter store output in your log file. The value is available in the environment variable $MONGODB_URI and can be used to pass along to any command or script that needs this value. CodeBuild is merely doing a best-effort masking of log statement which may contain sensitive information.

Also, if you must see the value printed out, you may assign $MONGODB_URI to another variable within your buildspec and print that out.


Cloudformation issue with target groups and ASG

Resources:
    TargetGroup:
        Type: AWS::ElasticLoadBalancingV2::TargetGroup
        Properties:
          VpcId:
            Fn::ImportValue:
              !Sub "${Prefix}-VpcId"
          TargetType: instance
          Port: 443
          Protocol: HTTPS
    AutoScalingGroup:
        Type: AWS::AutoScaling::AutoScalingGroup
        Properties:
          AutoScalingGroupName: Autoscaling Group Alpha
          Cooldown: "120"
          DesiredCapacity: "1"
          LaunchConfigurationName: AlphaLaunchConfiguration
          MaxSize: "1"
          MinSize: "1"
          Tags:
            -
              Key: Name
              Value: Alpha ASG
              PropagateAtLaunch: true
          TargetGroupARNs:
            - Ref: TargetGroup

As the error message states, TargetGroupARNs value should be of type List.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-targetgrouparns


THe issue was related to permission capabilities. Thanks


Your first step is to setup and configure the Route53 service from the rancher catalog.

After that, the load-balancer will automatically update its DNS in Route53.


How should I run a multicontainer app/platform with nginx behind traefik?

Although I agree with @BruceBecker's comment, i.e. running two different rproxies, it is possible to run traefik in front of nginx. At the previous job we configured several servers that were running on nginx behind traefik.

Ensure that each microservice has the correct labels. Note that Traefik 2.0 changed significantly.


503 Service Temporarily Unavailable using Kubernetes

i'm guessing the problem is that the name of the service port is not configured, and in the ingress you specified the http name and it can not find it.
name your port http , or give the port that you want to use (80) to the servicePort in the ingress file


What are some clear benefits of running master/agent configuration management?

I've found an interesting discussion on the topic configuration management pull vs push topology

One contributor writes that pull systems (master/agent) offer more flexibility and are easier to scale.


How to log outcome from multiple hosts in Ansible?

This question has two approaches, and both has answers on SO:

Convert all Ansible output to JSON with json stdout callback and parse it with your favourite tools. The eos_command output for each device will be under different host key in the resulting json.
See this and this.
Make run_once task to collect data from other hosts, like this:

- debug:
    msg: "{{ ansible_play_hosts | map('extract', hostvars, 'result') | map(attribute='stdout_lines') | list }}"
  run_once: yes

See this.


What is the difference between php-cli and php-fpm, why 2 php variants and why choose one or the other?

They are different tools for different purposes:

php-cli is the standalone tool for running a PHP scripts in the same way you would run it from the command line: php somescript.php.  It seems you can also use it to base other images from.

This variant contains the PHP CLI tool with default mods. If you need a web server, this is probably not the image you are looking for. It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as a base from which to build other images.

php-fpm is the FastCGI server implementation of PHP which you would use with a FastCGI compliant web server such as Apache or Nginx.

This variant contains PHP-FPM, which is a FastCGI implementation for PHP. See the PHP-FPM website for more information about PHP-FPM.

You also have two other variants:

php-apache which is just Apache configured with mod_php
php-alpine is the smallest possible Docker image based upon Alpine Linux, albeit with some compatibility issues.

You can see all of the Dockerfiles on GitHub and whilst it doesn't appear that php-fpm is based upon php-cli practically they are likely to be very similar and php-fpm is probably a superset of php-cli.


I have also asked the AWS Support folks for help too, as Lambda's security was crucial for us [HIPAA compliancy].

This was their response:

  Lambda has two options, you can have your lambda functions without a
  VPC or you could place your lambda function within a VPC. If you want
  to put your lambda function within VPC, then the most important thing
  to note is lambda function must be placed in private subnet. If you
  want to access internet resources from Lambda or make any AWS SDK API
  Calls, then you need to have the internet access for which you need to
  have a NAT Gateway or NAT Instance setup for the VPC, where your
  lambda function resides and have the private subnet point to the NAT
  Gateway. When lambda functions are placed within public subnet ( with
  an IGW route), then the lambda function won't work.

So, a lambda function can be put into a private subnet of a VPC, but not inside a public subnet. And for being able to create instances and ssh'ing into them, from this Lambda, add the private subnet to the security group of the public subnet [or another hacky way, is to add the VPC's IP range to the security group].


How to structure DevOps related code and configs in a code repository?

I can answer bout docker, one of the best practice to use docker is to keep the docker file and the compose files in the same repository of the project, so wherever you clone the project you can build the docker image, and it is good to keep multiple versions of docker compose files for example (prod, staging, dev) so you can build the image and run the container with specific option for each env for example for dev machine you can use specific network and run more dependencies container or whatever.


Should I clean up my local machine before installing Docker?

Ultimately, as long as you have the disk space and RAM, you can run both in parallel with very few issues.

The biggest problem you may run into is that the default ports of the applications you run in a container may already be in use.  For example SQL server install runs on port 1433 by default.  If you try to run a SQL server container and do not map a different port on the host machine to 1433 on the container you will run into errors.  This is easy enough do though:

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=" -p 1431:1433 -d mcr.microsoft.com/mssql/server:2019-CU3-ubuntu-18.04

In the above example, MSSQL server will run on port 1431 and map to port 1433 in the container.  On your local machine you can access either your SQL server on port 1433 or the container SQL server on 1431.


jenkins credentials not used as expected when cloning git repo

The Gitlab plugin requires an auth token, you're trying to use an SSH key.

https://github.com/jenkinsci/gitlab-plugin#global-plugin-configuration

"PLEASE NOTE: This auth configuration is only used for accessing the GitLab API for sending build status to GitLab. It is not used for cloning git repos. The credentials for cloning (usually SSH credentials) should be configured separately, in the git plugin."


What is the best way to put on a repository for a docker image with multiple tags?

If you think of the Dockerfile as a kind of "bill of materials" for your containers, then there is the following consideration:

container environments are mostly same - you can use many Dockerfiles in same folder, or even just one Dockerfile with parametrized builds
container environments are or might divert in terms of further "local assembly parts" configuration files, scripts etc., then you are better off with using separate folders.

Branching has rather to do with developing versions, or also independent features which then should be anyway merged together. See Git flow on this topic.

As a reference project laoyut, check for example the GitHub project linked from Tomcat Docker Hub site. Here you find different folders, even with single Dockerfiles.

See also this TeX-Docker distro:

https://hub.docker.com/r/raabf/texstudio-versions/

Side note: strange enough there is no "official" Docker distro for TeX, while many other well established open source communities have this library entry.


How to use port 5433 with Azure managed postgres

You can not change the port that Azure Database for PostgreSQL server uses. Connections to your Azure Database for PostgreSQL server communicate over port 5432. When you try to connect from within a corporate network, outbound traffic over port 5432 might not be allowed by your network's firewall. If so, you can't connect to your server unless your IT department opens port 5432.

If you truly need to use port 5433 you can install PostgreSQL on an Azure VM (IaaS).


In jenkins, how to re-run old builds of a job?

If the job is a jenkins pipeline, open the failed build and select the Replay option. Otherwise select the Rebuild option.

Sample Replay Option:

Sample Rebuild Option:


Switching between blue / green services in Kubernetes using UI page

Jenkins allows you to construct a CI/CD pipeline that meets your requirements. Here's an example of a pipeline using an input step to decide which environment to deploy to. All you would need to do is wire the stages together.

Here's an example of the pipeline code (declarative) used to create the example:

pipeline {
  agent any
  stages {
    stage('Deploy Staging') {
      steps {
        echo 'deploying to staging'
      }
    }
    stage('Test Staging') {
      steps {
        echo 'testing staging env'
      }
    }
    stage('Blue/Green?') {
      steps {
        input(message: 'Blue/Green?', id: 'deploy', ok: 'deploy?', parameters: [choice(choices: 'blue\ngreen', description: 'Select an environment', name: 'PROD_ENV')])
      }
    }
  }
}

Other tools that provide similar capabilities are Rundeck, Stackstorm or Ansible Tower.


What is the most common server permission policy for web application deployment?

According to the Nginx docs you can configure the user which runs Nginx. By default the user is nobody.

You can read this Ask Ubuntu question that explains how to configure Apache2 to run as a different user. The default user running Apache2 is www-data.

A common strategy, and one I think may work for your goal, is to assign the users that need to deploy the application to a common group. For instance, if you added them to the www-data group:

sudo adduser <user> www-data

You can see which groups is a member of with:

groups <user>

Then change the group of the directory where users will run migrate.php:

sudo chgrp -R www-data /var/www/demoapp

Then assign the group read/write/execute permissions to the directory:

sudo chmod -R g+rwx /var/www/demoapp/

The users in this group should now be able to write and run scripts within the demoapp/ directory.


How can I find out when Terraform will support newly released AWS services?

Terraform is an open source project that is managed on GitHub. You can follow existing issues or open a new one if your enhancement request isn't found. See their issue list at: https://github.com/hashicorp/terraform/issues

As Martin Atkins points out in the comments, the plugin specific features are each developed and tracked in their respective repositories. Here's the one for AWS: https://github.com/terraform-providers/terraform-provider-aws


Judging by the documentation, the result of your run multiple commands on remote nodes block is an array. You're writing only the first item of that array with {{ print_output.stdout[0] }}. The [0] here is the pointer to the first element of the array. Try removing it and see if the result fits your needs.
UPD: Use this to write several strings into a file:
- name: save output to a file
  lineinfile:
    create: yes
    line: "{{item}}"
    path: ./output/{{ inventory_hostname }}.txt
  with_items: "{{ print_output.stdout }}"


s3 static hosting not redirecting my request

The blog post you're following is somewhat obsolete, or at least incomplete in today's environment, because S3's web site hosting endpoints don't support HTTPS at all.  The request will time out.

To enable HTTPS on a custom domain pointing to a bucket, you need a certificate issued in the us-east-1 region of Amazon Certificate Manager (or you can purchase one from a third party and load it into ACM there) and then a CloudFront distribution to handle the HTTPS and forward the request to S3.

See How do I use CloudFront to serve a static website hosted on Amazon S3?  Note that these instructions are still applicable even though you aren't, strictly speaking, "hosting a static web site" on S3.  You're still using the static hosting feature and this is how you enable HTTPS.


How can I expose port 3389 using a secure approach

You can add, if it's not already the case, a frontal, rev proxy, to prevent direct access to the TS. This could also add an authentication layer...
I am not a lot in Microsoft stuff so this is really a basic advice.
If TS allow the use of pubkey instead of passwd this would be a good addition too.


Jenkins Pipeline GitLab API Credentials Management

Are you doing the building using web hooks or just polling? I think it's inevitable that you'll have to create a Jenkins account to manage API access. This Gitlab link explains how to use the Jenkins Gitlab plugin to manage building when changes are pushed. You'll see they use a Jenkins account as well. This is common practice in my organization to remove single points of failure, such as a departing employee.


Is it possible to use multiple if statements in a salt state?

You can have more than if statement per state. The issue is that your conditional doesn't seem actually be checking the hostname, it's just passing everything. Try this:

{% if grains['host'] in ['dev-server2', 'test-server2'] %}


How do I set a base domain for gitlab's auto devops?

https://docs.gitlab.com/ee/topics/autodevops/quick_start_guide.html#install-ingress

Go back to the cluster page on GitLab, and go to the Details tab. Add
your Base domain. For this guide, use the domain .nip.io.
Click Save changes.

I think this option only showed up AFTER I installed the ingress.


Understanding DevOps opponents on stakeholder level beyond typical examples

People resist change, particularly when it's perceived to be outside of their control.  Organizations that have been around a while naturally devolve into silos, and organizational structures often reflect that (e.g., database administration teams often manage all of the databases for the org, even though they are often distanced from the business purpose of those databases).  DevOps has the potential to disrupt silos, and may cause realignment of org structures.  For example, you could have product teams composed of developers and operations people (like DBA's) that focus solely on a single product.

For some mid-level managers, that could threaten their sense of job security.  The benefit of a DevOps mindset is that it encourages everyone to understand the value stream to the customer, and the relationship of all the roles in delivering that value.  IT could mean that the manager of DBA's suddenly has to understand development processes because they're now managing a product team.


How not to break Jenkins Pipeline when adding new parameters (declarative pipeline)

Here is a solution which seems almost like a workaround that I have found which works for declarative pipelines.

First, we need to add some code at the start of our file before we enter into the pipeline{} section. The code we add will run before the pipeline section, and will check if the env/param exists, and if it does will take the value, otherwise we will give it a default and assign that to a variable that we can access during the pipeline. Then we need to change our pipeline code references from the parameter to the variable.

Below is a sample that defines an arbitrary parameter, and also an agent label to be used. As you can see, the variables can be used during stages and also during procedures such as determining the agent.

def MY_VARIABLE = null
if (env.my_parameter) {
  MY_VARIABLE = env.my_parameter
} else {
  MY_VARIABLE = "my default value"
}
echo "MY VARIABLE IS SET: ${MY_VARIABLE}"

def MY_AGENT = null
if (env.my_agent) {
  MY_AGENT = env.my_agent
} else {
  MY_AGENT = "agent_label"
}
echo "MY AGENT IS SET: ${MY_AGENT}"

pipeline {
    parameters {
        string(defaultValue: "my default value", description: 'This is my parameter', name: 'my_parameter') ;
        string(defaultValue: "agent_label", description: 'This is my agent label', name: 'my_agent') ;
    }
    agent {
      label "${MY_AGENT}"
    }
    stages {
        stage('my stage') {
            steps {
                sh 'echo ${MY_VARIABLE}'
            }
        }
    }
}


def suiteRunId = UUID.randomUUID().toString() worked at the top of the Jenkinsfile.

Thanks all for your answers.


Kubernetes nginx ingress controller - one pr. environment?

I would suggest at least one for production and one for npe.  The nginx-ingress helm chart appears to only support setting a scope to all namespaces or just one namespace (a frustrating limitation not in other ingress controller options):

https://github.com/helm/charts/tree/master/stable/nginx-ingress

controller.scope.namespace


How do I replicate a Jenkins setup via automation?

Jenkins itself can be configured in many ways. The same applies for Jenkins jobs.

Probably the most easy way is to configure Jenkins manually and replicate its setup by backing up and restoring. Unfortunately, this means dealing with copying XML files.

But fortunately there are other ways.

init.groovy.d directory

When Jenkins starts, it checks init.groovy.d directory and executes Groovy scripts there. To enforce locale I'm using a snippet.

// Copied from /var/lib/jenkins/init.groovy.d/set-locale.groovy
def pluginWrapper = jenkins.model.Jenkins.instance.getPluginManager().getPlugin('locale')
def plugin = pluginWrapper.getPlugin()

plugin.setSystemLocale('en_US')

Another example of setting up Jenkins by Groovy scripts is located on https://gist.github.com/hayderimran7/50cb1244cc1e856873a4 - I'm using that snippet to setup local Jenkins users on test machines.

Generating jobs

Jenkins Job builder

One approach of generating jobs is Jenkins Job Builder, which creates XML definitions jobs from YAML configurations. I don't like this approach personally as I consider it to be a bit hackish - looks like "generation of XML files without knowing what it means". But some big players are using this approach - JJB originates from Openstack.

Job DSL plugin

Another approach is Job DSL plugin (plugin page). There is special job type called seed job which creates other jobs from Groovy DSL. The seed job can pull the others job description in from Git, so you can easily version them, reuse them, etc.

I'm using personally this approach, as it was available when I started with Jenkins. I've even found some Jenkins tool to create DSL sources from existing jobs, but I'm not able to find it again.

To provide a seed job, you can still use the init.groovy.d approach.

Pipelines

The currently recommended approach is probably related to Jenkins Pipelines. You can store your building pipelines in the source VCS repository next to your code.

I haven't explored this very much, but I'm planning to do.